{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# feature_importance_permutation: Estimate feature importance via feature permutation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A function to estimate the feature importance of classifiers and regressors based on *permutation importance*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> `from mlxtend.evaluate import feature_importance_permutation`    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *permutation importance* is an intuitive, model-agnostic method to estimate the feature importance for classifier and regression models. The approach is relatively simple and straight-forward:\n",
    "\n",
    "1. Take a model that was fit to the training dataset\n",
    "2. Estimate the predictive performance of the model on an independent dataset (e.g., validation dataset) and record it as the baseline performance\n",
    "3. For each feature *i*:\n",
    "   - randomly permute feature column *i* in the original dataset\n",
    "   - record the predictive performance of the model on the dataset with the permuted column \n",
    "   - compute the feature importance as the difference between the baseline performance (step 2) and the performance on the permuted dataset\n",
    "\n",
    "Permutation importance is generally considered as a relatively efficient technique that works well in practice [1], while a drawback is that the importance of correlated features may be overestimated [2].\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "- [1] Terence Parr, Kerem Turgutlu, Christopher Csiszar, and Jeremy Howard. *Beware Default Random Forest Importances* (https://parrt.cs.usfca.edu/doc/rf-importance/index.html)\n",
    "- [2] Strobl, C., Boulesteix, A. L., Kneib, T., Augustin, T., & Zeileis, A. (2008). Conditional variable importance for random forests. BMC bioinformatics, 9(1), 307."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 1 -- Feature Importance for Classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following example illustrates the feature importance estimation via permutation importance based for classification models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.model_selection import train_test_split\n",
    "from mlxtend.evaluate import feature_importance_permutation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Generate a toy dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_classification\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# Build a classification task using 3 informative features\n",
    "X, y = make_classification(n_samples=10000,\n",
    "                           n_features=10,\n",
    "                           n_informative=3,\n",
    "                           n_redundant=0,\n",
    "                           n_repeated=0,\n",
    "                           n_classes=2,\n",
    "                           random_state=0,\n",
    "                           shuffle=False)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.3, random_state=1, stratify=y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Feature importance via random forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we compute the feature importance directly from the random forest via *mean impurity decrease* (described after the code section):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy: 100.0\n",
      "Test accuracy: 95.06666666666666\n",
      "[0.283357   0.30846795 0.24204291 0.02229767 0.02364941 0.02390578\n",
      " 0.02501543 0.0234225  0.02370816 0.0241332 ]\n"
     ]
    }
   ],
   "source": [
    "forest = RandomForestClassifier(n_estimators=250,\n",
    "                                random_state=0)\n",
    "\n",
    "forest.fit(X_train, y_train)\n",
    "\n",
    "print('Training accuracy:', np.mean(forest.predict(X_train) == y_train)*100)\n",
    "print('Test accuracy:', np.mean(forest.predict(X_test) == y_test)*100)\n",
    "\n",
    "importance_vals = forest.feature_importances_\n",
    "print(importance_vals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several strategies for computing the feature importance in random forest. The method implemented in scikit-learn (used in the next code example) is based on the Breiman and Friedman's CART (Breiman, Friedman, \"Classification and regression trees\", 1984), the so-called *mean impurity decrease*. Here, the importance value of a features is computed by averaging the impurity decrease for that feature, when splitting a parent node into two child nodes, across all the trees in the ensemble. Note that the impurity decrease values are weighted by the number of samples that are in the respective nodes. This process is repeated for all features in the dataset, and the feature importance values are then normalized so that they sum up to 1. In CART, the authors also note that this fast way of computing feature importance values is relatively consistent with the permutation importance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's visualize the feature importance values from the random forest including a measure of the *mean impurity decrease* variability (here: standard deviation):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAWiUlEQVR4nO3dfbRddX3n8feHQBQCHSykIiQ8qIzIWIs0RR0dm6pMwSfUOgpWHRwpQ0dsnbGj6JpaXdYOznKszgyaQUV0+cBQUVeqqfhUdVDRBEUlChpDMBEDAYkIOsXAd/7YO+3meh/OhXvPDb/7fq11VvbD7+7vb599zufss8/eO6kqJEn3fXstdAckSXPDQJekRhjoktQIA12SGmGgS1IjDHRJaoSB3rgkr0vy/oXux54gnfckuSXJ1xa6P7OV5LYkD17ofmjPZaAvgCRbkvyif4NuT3Jhkv0Xul/3RpLVSe7q12n342/HWP/IJJVk72maPR44EVhRVSfcy3qnJ7ns3ixjtqpq/6raPM6aU+mf64cudD90dwb6wnl6Ve0PHAc8Cnj1wnZnTlzfh87ux9Nnu4AkS+ajY70jgC1Vdfs81hjJDB88e6z7ar8XCwN9gVXVduBSumAHIMk5SX6Q5GdJvpPkWYN5pye5LMmb+0MH1yY5eTD/qCRf6P/208DBw3pJnpFkY5KdST6f5OGDeVuS/Ock30pye5J3J3lgkr/rl/eZJA+Y7TomeXhfa2df+xmDeRcmeUeSdUluB34vyaFJLkmyo1+/Pxm0PyHJhiS3JrkhyVv6WV/s/93Zfzt47IQ+vAR4F/DYfv7r++lPS3Jl37cvJ3nkTNuhf87WDJa1s5/++SRnTNxWg/FK8tIk3we+P1P9SZ7Hf9wr7p+3t/fb5rYkX0pySJK39q+Lq5M8avC3W5K8ul+PW/pDT/cfzP+jJJuS/CTJ2iSHTtXvJLuf62/2tZ+X5AFJPt5vs1v64RWDZXw+yRv6fv4syaeSHDyY//h+/Xcm2Zrk9H76/frX+g/77b0myb5TPUeLXlX5GPMD2AI8uR9eAXwbeNtg/r8BDqX7wH0ecDvwoH7e6cAvgT8ClgB/DFwPpJ//FeAtwP2AJwA/A97fz/vn/bJOBPYBXglsApYO+nU58EDgMOBG4Ot03yDuB3wO+Isp1mk1sG2S6fv0NV4DLAWe2PfpYf38C4GfAo/r13c/4ArgtX37BwObgd8frN8L++H9gcf0w0cCBew9zfN+OnDZYPz4fh0f3T+X/7Z/Du434na4bMLyPw+cMU29Aj4N/Dqw70z1J+l/AQ8dPG83Ab8N3L/fNtcCL+qX9ZfA3094zV0FrOzrfwn4y37eE/tlHd9v5/8JfHGqfk/sSz9+EPAH/fY7APgb4GMTnpsf0L0G9+3Hz+3nHd6/Jk6je70cBBzXz3srsLavfQDwt8B/Xej38J76WPAOLMZH/+a6rX8RF/BZ4MBp2l8JnNIPnw5sGszbr1/GIf0bYxewbDD/g/xToP85cPFg3l7Aj4DVg3794WD+JcA7BuMvG75JJ/RxNXAXsHPweC7wr4DtwF6Dth8CXtcPXwi8bzDv0cAPJyz71cB7+uEvAq8HDp7Q5khmH+jvAN4woc01wO+OuB3uSaA/8V7Unxjo75ywbb47GP9NYOeE19xZg/GnAD/oh98N/LfBvP3pdhqOnKzfE/syRV+PA26Z8Nz8l8H4fwA+Odi+H51kGaH7EH3IYNpjgWvn4n3Y4sNDLgvnmVV1AF0QHsPg0EiSFw2+hu8EHsHdD51s3z1QVT/vB/en25u8pe5+jPi6wfChw/GqugvYSrc3vtsNg+FfTDI+3Y+311fVgYPHxX3NrX2tYZ+GNbcOho8ADt297v36v4buWwPAS+j28q5Osj7J06bpz0yOAF4xodbKvs+jbId7YuK6Tll/BLPdVsPa1w3qTHxd3AbczNTb6Fck2S/J/05yXZJb6T54D8zdfxPZPhj++aB/K+n23idaTv+NbfD8fLKfrkn4A8cCq6ovJLkQeDPwzCRHAO8EngR8paruTHIl3d7KTH4MPCDJskGoH063NwXdoZnf3N04SejeTD+ai3WZwvXAyiR7DUL9cOB7gzbDW35updsDO3qyhVXV94HTkuwFPBv4cJKDJixjVFuBN1bVGyfOGGE7TFbvdroA2u2QyVZhlPrzZOVg+HC6bUP/7xG7ZyRZRnfYY/i6mOn5fQXwMODRVbU9yXHANxjtdbsVmOyso5voPpj+RVXN52u0Ge6h7xneCpzYvwmW0b15dgAkeTHdnuGMquo6YAPw+iRLkzweGJ5pcjHw1CRPSrIP3ZvwH4Avz9F6TOardEH3yiT7JFnd9+miKdp/Dbg1yauS7JtkSZJHJPkdgCQvSLK8/3DY2f/NnXTP1110x9xH9U7grCSPTmdZkqcmOYCZt8MNwIokSwfTrgSe3e+tPpTu28Q9rT8fXppkRZJfp/vW83/66R8EXpzkuCT3A/4K+GpVbZlmWTdw9+f6ALrw3dkv/y9m0a8PAE9O8twkeyc5KMlx/TZ+J/DXSX4DIMlhSX5/FsteVAz0PUBV7QDeB/x5VX0H+O90P/7dQLdH/aVZLO75dMehf0L3pnrfoM41wAvofvS6iS5Yn15Vd8zBakyqX/YzgJP7mm8HXlRVV0/R/s6+X8fR/ch3E93ZKf+sb3ISsDHJbcDbgFOr6v/1h57eCHyp/3r+mBH6toHux+X/BdxC9+Pt6f28mbbD54CNwPYkN/XT/hq4o2//Xrqgukf158kHgU/R/ci8me6HU6rqs3S/r1xC9y3vIcCpMyzrdcB7++f6uXQ7JfvSba/L6Q6NjKSqfkh3TP8VdK/bK4Hf6me/iu55ubw/lPMZum8CmsTuMyMkNSzJFrofbD+z0H3R/HEPXZIaMVKgJzkpyTXpLjw4Z5L5q5P8tD8j4Mokr537rkqSpjPjWS79aUfn0V2Msg1Yn2Rtf4xx6P9W1b05hUzSPKmqIxe6D5p/o+yhn0B3Icvm/geui4BT5rdbkqTZGuU89MO4+0UF2+jOopjosUm+SXdO659V1caJDZKcCZwJsGzZst8+5phjZt9jSVrErrjiipuqatKLq0YJ9MkuDJh4aszXgSOq6rYkTwE+BvzKhSFVdT5wPsCqVatqw4YNI5SXJO2W5Lqp5o1yyGUbd7/CbAX/dIUZAFV1a3+5MFW1DthneCc1SdL8GyXQ1wNHp7st61K6Cw7WDhuku21n+uET+uXePNedlSRNbcZDLlW1K8nZdPfsXgJcUFUbk5zVz18DPAf44yS76C7/PbW8YkmSxmrBrhT1GLokzV6SK6pq1WTzvFJUkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqxEiBnuSkJNck2ZTknGna/U6SO5M8Z+66KEkaxYyBnmQJcB5wMnAscFqSY6do9ybg0rnupCRpZqPsoZ8AbKqqzVV1B3ARcMok7V4GXALcOIf9kySNaJRAPwzYOhjf1k/7R0kOA54FrJluQUnOTLIhyYYdO3bMtq+SpGmMEuiZZFpNGH8r8KqqunO6BVXV+VW1qqpWLV++fMQuSpJGsfcIbbYBKwfjK4DrJ7RZBVyUBOBg4ClJdlXVx+aik5KkmY0S6OuBo5McBfwIOBV4/rBBVR21ezjJhcDHDXNJGq8ZA72qdiU5m+7slSXABVW1MclZ/fxpj5tLksZjlD10qmodsG7CtEmDvKpOv/fdkiTNlleKSlIjDHRJaoSBLkmNMNDHZPXq1axevXqhuyGpYQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIxZdoHvXQ0mtWnSBLkmtMtAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwz0RcBz76XFwUCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNWLvURolOQl4G7AEeFdVnTth/inAG4C7gF3Ay6vqsjnu66SOPOcTs2q/ffPNs/67Lec+dVY1JGkhzBjoSZYA5wEnAtuA9UnWVtV3Bs0+C6ytqkrySOBi4Jj56LAkaXKjHHI5AdhUVZur6g7gIuCUYYOquq2qqh9dBhSSpLEaJdAPA7YOxrf10+4mybOSXA18Avh3ky0oyZlJNiTZsGPHjnvSX0nSFEYJ9Ewy7Vf2wKvqo1V1DPBMuuPpv/pHVedX1aqqWrV8+fJZdVSSNL1RAn0bsHIwvgK4fqrGVfVF4CFJDr6XfZMkzcIogb4eODrJUUmWAqcCa4cNkjw0Sfrh44GlwM1z3VlJ0tRmPMulqnYlORu4lO60xQuqamOSs/r5a4A/AF6U5JfAL4DnDX4klSSNwUjnoVfVOmDdhGlrBsNvAt40t12TJM2GV4pKUiMMdElqxEiHXDS52dw+wFsOSJpv7qFLUiMMdElqhIEuSY0w0CWpEYvuR9FDnn/uzI0k6T7IPXRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNWLR3ZyrBbP5X4/A/y1JWizcQ5ekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjRgp0JOclOSaJJuSnDPJ/D9M8q3+8eUkvzX3XZUkTWfGQE+yBDgPOBk4FjgtybETml0L/G5VPRJ4A3D+XHdUkjS9UfbQTwA2VdXmqroDuAg4Zdigqr5cVbf0o5cDK+a2m5KkmYwS6IcBWwfj2/ppU3kJ8HeTzUhyZpINSTbs2LFj9F5KkmY0SqBnkmk1acPk9+gC/VWTza+q86tqVVWtWr58+ei9lCTNaJT/U3QbsHIwvgK4fmKjJI8E3gWcXFU3z033JEmjGmUPfT1wdJKjkiwFTgXWDhskORz4CPDCqvre3HdTkjSTGffQq2pXkrOBS4ElwAVVtTHJWf38NcBrgYOAtycB2FVVq+av25KkiUY55EJVrQPWTZi2ZjB8BnDG3HatLYc8/9yF7oKkxnmlqCQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiNGutui7tu806O0OLiHLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRowU6ElOSnJNkk1Jzplk/jFJvpLkH5L82dx3U5I0k71napBkCXAecCKwDVifZG1VfWfQ7CfAnwDPnI9OSpJmNsoe+gnApqraXFV3ABcBpwwbVNWNVbUe+OU89FGSNIJRAv0wYOtgfFs/bdaSnJlkQ5INO3bsuCeLkCRNYZRAzyTT6p4Uq6rzq2pVVa1avnz5PVmEJGkKowT6NmDlYHwFcP38dEeSdE+NEujrgaOTHJVkKXAqsHZ+uyVJmq0Zz3Kpql1JzgYuBZYAF1TVxiRn9fPXJDkE2AD8GnBXkpcDx1bVrfPXdUnS0IyBDlBV64B1E6atGQxvpzsUI0laIF4pKkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBrnmzevVqVq9evWjqSgvNQJfm0GL8EHOd9xwGuiQ1Yu+F7oDuW4485xMjt92++eZZ/82Wc5+6IHUXsvaeVlf3XQa6pF+x2D7EZvvhv6d+gHrIRZIa4R665s0hzz93UdVdyNqus8BAl6RZ21M/TDzkIkmNMNAlqREGuiQ1YqRAT3JSkmuSbEpyziTzk+R/9PO/leT4ue+qJGk6MwZ6kiXAecDJwLHAaUmOndDsZODo/nEm8I457qckaQaj7KGfAGyqqs1VdQdwEXDKhDanAO+rzuXAgUkeNMd9lSRNI1U1fYPkOcBJVXVGP/5C4NFVdfagzceBc6vqsn78s8CrqmrDhGWdSbcHD/Aw4Jq5WpFZOhi4aRHVXcjai63uQtZ2nRdH7SOqavlkM0Y5Dz2TTJv4KTBKG6rqfOD8EWrOqyQbqmrVYqm7kLUXW92FrO06L57aUxnlkMs2YOVgfAVw/T1oI0maR6ME+nrg6CRHJVkKnAqsndBmLfCi/myXxwA/raofz3FfJUnTmPGQS1XtSnI2cCmwBLigqjYmOaufvwZYBzwF2AT8HHjx/HV5TizUYZ+FPNy02NbZ53px1F6M6zylGX8UlSTdN3ilqCQ1wkCXpEYsqkBPckGSG5NctQC1p719wjzVXJnk75N8N8nGJH86jrqD+gcm+XCSq/s+PHZMdf80yVX9Or98HDX7uluSfDvJlUk2zPwXc1r7P/bre1WSDyW5/xhrL0nyjf56lHHVfFj/PO9+3DqObZ3k/km+luSb/fP9+vmuOStVtWgewBOA44Grxlx3CfAD4MHAUuCbwLFjqPsg4Ph++ADge+OoO6j/XuCMfngpcOAYaj4CuArYj+5H/88AR49pfbcAB4/ztdXXPQy4Fti3H78YOH2M9f8T8EHg4+Ne977+EmA73QU3810rwP798D7AV4HHLMR6T/ZYVHvoVfVF4CcLUHqU2yfMuar6cVV9vR/+GfBdujf/vEvya3QfoO/u699RVTvHUPrhwOVV9fOq2gV8AXjWGOoutL2BfZPsTfdhNpbrQJKsAJ4KvGsc9abwJOAHVXXdfBeqzm396D79Y485s2RRBfoCOgzYOhjfxpiCdbckRwKPotujGIcHAzuA9/Rfx9+VZNkY6l4FPCHJQUn2ozudduUMfzNXCvhUkiv621yMp2jVj4A3Az8Efkx3HcinxlT+rcArgbvGVG8ypwIfGlex/hDTlcCNwKeralzvqRkZ6OMx0q0R5q14sj9wCfDyqrp1TGX3pju89Y6qehRwOzDvvx1U1XeBNwGfBj5Jd3hr13zX7T2uqo6nu/voS5M8YRxFkzyA7hvfUcChwLIkLxhD3acBN1bVFfNda5o+LAWeAfzNuGpW1Z1VdRzdFfEnJHnEuGrPxEAfjwW7NUKSfejC/ANV9ZFx1OxtA7YN9l4+TBfw866q3l1Vx1fVE+gOsX1/THWv7/+9Efgo3aG2cXgycG1V7aiqXwIfAf7lGOo+DnhGki10hxGfmOT9Y6g7dDLw9aq6Ycx16Q8hfh44ady1p2Kgj8cot0+Yc0lCdwz7u1X1lvmuN1RV24GtSR7WT3oS8J1x1E7yG/2/hwPPZgxfx5MsS3LA7mHgX9Md/hmHHwKPSbJfv82fRPd7ybyqqldX1YqqOpLuNf25qpr3bwYTnMZ4D7csT3JgP7wv3Yfp1eOqP5NR7rbYjCQfAlYDByfZBvxFVb17vuvWFLdPmO+6dHtQLwS+3R/zA3hNVa0bQ22AlwEf6D/ENjO+W0JckuQg4JfAS6vqljHUfCDw0S5P2Rv4YFV9cgx1qaqvJvkw8HW6w0vfYA+8LH2u9b+RnAj8+zGWfRDw3v4//tkLuLiqxna65ky89F+SGuEhF0lqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGvH/ATegAl2u/VuKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "std = np.std([tree.feature_importances_ for tree in forest.estimators_],\n",
    "             axis=0)\n",
    "indices = np.argsort(importance_vals)[::-1]\n",
    "\n",
    "# Plot the feature importances of the forest\n",
    "plt.figure()\n",
    "plt.title(\"Random Forest feature importance\")\n",
    "plt.bar(range(X.shape[1]), importance_vals[indices],\n",
    "        yerr=std[indices], align=\"center\")\n",
    "plt.xticks(range(X.shape[1]), indices)\n",
    "plt.xlim([-1, X.shape[1]])\n",
    "plt.ylim([0, 0.5])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the features 1, 0, and 2 are estimated to be the most informative ones for the random forest classier. Next, let's compute the feature importance via the permutation importance approach."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Permutation Importance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.26833333,  0.26733333,  0.261     , -0.002     , -0.00033333,\n",
       "        0.00066667,  0.00233333,  0.00066667,  0.00066667, -0.00233333])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imp_vals, _ = feature_importance_permutation(\n",
    "    predict_method=forest.predict, \n",
    "    X=X_test,\n",
    "    y=y_test,\n",
    "    metric='accuracy',\n",
    "    num_rounds=1,\n",
    "    seed=1)\n",
    "\n",
    "imp_vals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the `feature_importance_permutation` returns two arrays. The first array (here: `imp_vals`) contains the actual importance values we are interested in. If `num_rounds > 1`, the permutation is repeated multiple times (with different random seeds), and in this case the first array contains the average value of the importance computed from the different runs. The second array (here, assigned to `_`, because we are not using it) then contains all individual values from these runs (more about that later)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's also visualize the importance values in a barplot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEICAYAAACj2qi6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAaUklEQVR4nO3ce7QlZX3m8e9DNyi0GBXaKHRziRIRL0HsgEajRCWCN9BkFLxFo2EwamSio5iJEaPOqCujZCZGFgpelgJBUQcV78a7KA3BCyraItotAo2C2F4CDb/5o+pA9WGfc97uPmfvQ/f3s9ZZZ9dl1/tW1Vv17Hpr105VIUlSix0mXQFJ0m2HoSFJamZoSJKaGRqSpGaGhiSpmaEhSWq2qEIjyYlJ3j3peiwG6bw9yTVJvjbp+myuJBuS/N6k63FblOSPk1wy6XosRkmeluQTC7Bct3mjOUMjyWVJftOfBK5I8o4kdxhH5RZKkkOT3NSv09Tfh8ZY/j5JKsnSWWZ7KHAYsKKqDt7K8p6V5Itbs4zNVVV3qKpLx1nmTPptfc9J16NVVX2hqu416XrMt81th6OOk6p6T1X96XzXbTFt8/78tG7S9ZhJ65XG46vqDsCBwAOAly9Yjcbn8v7ENvX3+M1dQJIlC1Gx3t7AZVX1qwUso8kc4bZo3VbrvVi4/cbvNrHNq2rWP+Ay4FGD4TcAHxkMnwD8APgl8G3giYNpzwK+CPwTcA3wQ+CIwfR9gc/17/0k8C/AuwfTnwBcDFwLfBa497R6/XfgG8CvgFOB3wU+2i/vU8CdZ1inQ4F1M0y7d1/WtX3ZTxhMewfwFuDcvsxHAXsAZwPr+/X7m8H8BwOrgeuAK4E39uN/DBSwof978LQ6PAf4LXBjP/1V/fjHARf1dfsycP+59kO/PsNlXduP/yzw3On7ajBcwPOB7wM/nKv8EduxgHsOttu/9vtmA/Al4G7ASX27+C7wgGn79uX9elwDvB24/WD6XwFrgJ8D5wB7zFRv4PP9uF/1ZT8FuDPw4X6fXdO/XjFYxmeBV/f1/CXwCWD3wfSH9ut/LbAWeFY//nZ0bf3H/f4+Gdh5xLa5Xf/e+w7GLQd+A9yVae1zpn07w3Y/EXgf8G/9/BcCfzCYPlt7nXrvu+na7HP7bfGafn03AB8CdgPe089zPrBP//59+m29dNq2fC4zt8PHAv/RL2stcOLgvbc6Trh1O/2jvg6/6P//Uet+nO2cwGacXwbrfSxwOfBT4MXT9vdJ/bTL+9e3G5YLvAy4Anhv3w5uGqz3HnTnkq/QtZuf0p0rd5rW7o+ja/fXAG8GMu2Y+Q63tKGD5moPM7axzQkNYAXwTeCfB9P/S1/wDnQH5K+Auw9ORDf0FV4CPK/faOmnfwV4Y79RH9av0Lv7ab/fL+swYEfgpXQnip0G9Tqv35F7AlfRHSAP6Jf3GeCVmxMafTlrgL8DdgIe0dfpXoOT3y+Ah/TruwtwAfAP/fy/B1wKPHqwfs/oX98BeNBMB9eIujyLTQ+Og/p1PKTfln/Rb4PbNe6HL05b/meZOzQ+CdwF2Hmu8htC42rggcDt+33zQ+CZ/bJeA/z7tDb3LWBlX/6XgNf00x7RL+ugfj//X+DzM9V7el364d2AP+v33650B+oHp22bH9C1wZ374df10/bq28QxdO1lN+DAftpJdCF2l365HwL+1wzb5zTgtYPh5wMfm+EENuO+nSE0bgD+vK/fS/ptvWP//tna69R7j+rnnVr3NcA9gN+hO+F8j+4D01LgXcDb5wqNWdrhocD9+vLuTxe2R82yvJuX0W/na4Bn9HU5ph/eba792BgaTeeXQT3PAJb167OeW86b/9gv6650Hw6+DLx6UO5G4PX9cneeXpd+vgcCD+rXcx+6ADh+Wrv/MHAnuja6Hjh80H5+AvwhEOCedD0Zs7aHrQ2NDXQHSgGfBu40y/wXAUcOdvCawbRd+mXcrV+xjcCywfTTuSU0XgGcNZi2Q7/ihw7q9bTB9LOBtwyGX8jgRDCigdxEl9pTf08G/pgu7XcYzHsG/acfupPfuwbTDgF+PG3ZL+eWg+jzwKuY9umGLQuNt0w1tMG4S4CHN+6HLQmNR2xF+dND463T9s13BsP3o//kOdi3xw2GHwP8oH99KvCGwbQ70J3o9hlV7+l1maGuBwLXTNs2fz8Y/mtuOaG/HPjAiGWE7mR+j8G4B9NfpY2Y/1HApYPhLwHPHLTPkVfC0/ftiGknAudNO25+Ste252qvJzII4MG2+B+D4f8NfHQw/HjgopnaNXOExoj6nwS8aZbl3bwMurD42rT3f4Vbrvxm3I8jyt1km7MZ55dBPfcfTH8DcGr/+gfAYwbTHk3X9TxV7vVseiU96/7v5zl+2A778h86GD4LOKF//XHgRSOWMWt7mOmvtf/sqKr6VJKH053Yd6c70ZLkmcDf9hsOuoN498F7r5h6UVW/TjKc55ratM/+R3SfLqH7ZPWjwXtvSrKWLvWnXDl4/ZsRw7PdsL+8qlYMRyR5CrC2qm6aVqdhmWsHr/cG9khy7WDcEuAL/evn0H3K+G6SH9J1M314ljrNZm/gL5K8cDBuJ7rt1LIftsT0dZ2x/Aabu6+GZf9oUM4edJ/4AKiqDUl+RrePLhvx3ltJsgvwJuBwuq4qgF2TLKmqG/vhKwZv+fWgfivpTgLTLae/8uzbOHRBMtN9r88AOyc5pC/rQOADM9R3c/ftzevfHzfr6LZbMXt73eS9A1tznM2qX//XAfela0+3o7vya7HJOaI3/XidaT+22No2e78Z6jlszwDrq+q3s1Ukye/T9cqsomtnS+muEoY2t83Odf4aabNuulTV55K8g67f9qgkewNvBR4JfKWqbkxyEd3BMpefAndOsmwQHHvRNWzourGmNjrpjsSVdFcbC+VyYGWSHQbBsRfd5fiUGrxeS/dJcr9RC6uq7wPHJNkBeBLwviS7TVtGq7V03RmvnT6hYT+MKu9XdI1vyt1GrUJL+Qtk5eD1XnT7hv7/3lMTkiyj6yIatou5tu+LgXsBh1TVFUkOpOtXb2m3a+n6l6e7mu5Ecp+qmrON9ifzs+i6VK4EPlxVv5w+3xYeYzdvu77traDbbhuZpb1OVW2uus9i6jjehe4eBWzarkYt+3S6/vkjquq3SU7ilkCcqy6btIXeXsDHWis8z1bS3Z+bqsf0NnvxiGlw6/Uctd5voWujx1TVL5McT9cF2WItXffiqPFztYdb2ZLnNE4CDusPtGV0K7geIMmz6T4xzKmqfkR3k/hVSXZK8lC6S90pZwGPTfLIJDvSHej/SdcfuFC+StfwX5pkxySH9nU6c4b5vwZcl+RlSXZOsiTJfZP8IUCSpydZ3gfQtf17bqTbXjfR9SG2eitwXJJD+mc4liV5bJJdmXs/XAmsSLLTYNxFwJOS7NJ/HfU5W1H+Qnh+khVJ7kJ3j+nf+vGnA89OcmCS2wH/E/hqVV02y7KuZNNtvSvdCf7afvmv3Ix6vQd4VJInJ1maZLckB/b7+K3Am5LcFSDJnkkePcuyTqe7R/G0/vUoW3KMPTDJk/pv4hxPd9ycxxztdWtV1Xq68H56v+y/ZNOT1ah2uCvw8z4wDgaeOpg213FyLvD7SZ7a74unAAfQ9e1Pwiv64+k+wLO5pc2eAfx9kuVJdqe7hzDb82hXArsl+Z3BuF3pgnhDkv3p7g+3ehvwkiQP7I/de/YfRraoPWx2aPQN413AK6rq23R9nF+hW9H70fXNtnoqXb/az+kO3HcNyrkEeDrdjc6r6U7ej6+q6ze3zq36ZT8BOKIv81/p+pm/O8P8N/b1OpDuZuPVdDtoamcfDlycZAPwz8DRVfXbqvo18FrgS0muTfKghrqtpvtCwb/Q3exbQ9e/S8N++Azdp5wrklzdj3sTXV/qlcA76U6GW1T+Ajmd7tsul/Z/r+nr8Wm6+11n012t3gM4eo5lnQi8s9/WT6b74LMz3f46j834ZFpVP6a7x/JiunZ7EfAH/eSX0W2X85JcR/cNmxm/+19VUx9S9qD7Vs6oebbkGPt/dGF0DV2//5Oq6oaG9jof/oruW0c/A+7Dph/yRrXDvwb+Mckv6U6mZ03NPNdxUlU/o/tG34v78l4KPK6qrmYyPke3/z8N/FNVTT2E+Bq6D8jfoPsi0YX9uJH6880ZwKX9eu9B94WGp9LdW34rtwTSnKrqvXTb8fT+/R8E7rKl7WHqW0zSopHkMrqbp5+adF1ua5KcSHfT/+mTrsv2Isk+9N9Qq6qNE67OgltUPyMiSVrcmkMjyeFJLkmyJskJI6YfmuQXSS7q//5hfqsqSZq0pu6pdD+X8T26B+3W0T15eUzf3zo1z6HAS6rqcQtSU0nSxLVeaRxM95Depf3N4jOBIxeuWpKkxaj1OY092fTBlXV033qa7sFJvk73HeSXVNXF02dIcizdb7SwbNmyB+6///6bV2NJ2s5dcMEFV1fV8kmU3Roaox4kmt6vdSGwd/+E7mPovtZ1q4dGquoU4BSAVatW1erVq9trK0kiyfQn4cemtXtqHZs+oTv1hOnNquq6qtrQvz4X2LF/kEWStI1oDY3zgf2S7Ns/zXk03a953izJ3ZLuR3f6Jzt3oHvgRpK0jWjqnqqqjUleQPdriUuA06rq4iTH9dNPpvsdlOcl2Uj3Ew1Hl08OStI2ZaJPhHtPQ5I2X5ILqmrVJMr2iXBJUjNDQ5LUzNCQJDUzNCRJzQwNSVIzQ0OS1MzQkCQ1MzQkSc0MDUlSM0NDktTM0JAkNTM0JEnNDA1JUjNDQ5LUzNCQJDUzNCRJzQwNSVIzQ0OS1MzQkCQ1MzQkSc0MDUlSM0NDktTM0JAkNTM0JEnNDA1JUjNDQ5LUzNCQJDUzNCRJzQwNSVIzQ0OS1MzQkCQ1MzQkSc0MDUlSM0NDktTM0JAkNWsOjSSHJ7kkyZokJ8wy3x8muTHJn89PFSVJi0VTaCRZArwZOAI4ADgmyQEzzPd64OPzWUlJ0uLQeqVxMLCmqi6tquuBM4EjR8z3QuBs4Kp5qp8kaRFpDY09gbWD4XX9uJsl2RN4InDybAtKcmyS1UlWr1+/fnPqKkmasNbQyIhxNW34JOBlVXXjbAuqqlOqalVVrVq+fHlj8ZKkxWBp43zrgJWD4RXA5dPmWQWcmQRgd+AxSTZW1Qe3tpKSpMWhNTTOB/ZLsi/wE+Bo4KnDGapq36nXSd4BfNjAkKRtS1NoVNXGJC+g+1bUEuC0qro4yXH99FnvY0iStg2tVxpU1bnAudPGjQyLqnrW1lVLkrQY+US4JKmZoSFJamZoSJKaGRqSpGaGhiSpmaEhSWpmaEiSmhkakqRmhoYkqZmhIUlqZmhIkpoZGpKkZoaGJKmZoSFJamZoSJKaGRqSpGaGhiSpmaEhSWpmaEiSmhkakqRmhoYkqZmhIUlqZmhIkpoZGpKkZoaGJKmZoSFJamZoSJKaGRqSpGaGhiSpmaEhSWpmaEiSmhkakqRmhoYkqZmhIUlq1hwaSQ5PckmSNUlOGDH9yCTfSHJRktVJHjq/VZUkTdrSlpmSLAHeDBwGrAPOT3JOVX17MNungXOqqpLcHzgL2H++KyxJmpzWK42DgTVVdWlVXQ+cCRw5nKGqNlRV9YPLgEKStE1pDY09gbWD4XX9uE0keWKS7wIfAf5y1IKSHNt3X61ev3795tZXkjRBraGREeNudSVRVR+oqv2Bo4BXj1pQVZ1SVauqatXy5cubKypJmrzW0FgHrBwMrwAun2nmqvo8cI8ku29F3SRJi0xraJwP7Jdk3yQ7AUcD5wxnSHLPJOlfHwTsBPxsPisrSZqspm9PVdXGJC8APg4sAU6rqouTHNdPPxn4M+CZSW4AfgM8ZXBjXJK0Dcgkz+urVq2q1atXT6x8SbotSnJBVa2aRNk+ES5JamZoSJKaGRqSpGaGhiSpmaEhSWpmaEiSmhkakqRmhoYkqVnTE+G3Bfuc8JEFXf5lr3vsgi5fkm4LtpnQmJSFDiswsCQtHnZPSZKaeaVxG2aXnKRx80pDktTM0JAkNTM0JEnNDA1JUjNDQ5LUzNCQJDUzNCRJzQwNSVIzQ0OS1MzQkCQ1MzQkSc0MDUlSM0NDktTM0JAkNTM0JEnNDA1JUjNDQ5LUzNCQJDUzNCRJzQwNSVIzQ0OS1MzQkCQ1MzQkSc2aQyPJ4UkuSbImyQkjpj8tyTf6vy8n+YP5raokadKaQiPJEuDNwBHAAcAxSQ6YNtsPgYdX1f2BVwOnzGdFJUmT13qlcTCwpqourarrgTOBI4czVNWXq+qafvA8YMX8VVOStBi0hsaewNrB8Lp+3EyeA3x01IQkxyZZnWT1+vXrG4uXJC0GraGREeNq5IzJn9CFxstGTa+qU6pqVVWtWr58eWPxkqTFYGnjfOuAlYPhFcDl02dKcn/gbcARVfWzra+eJGkxab3SOB/YL8m+SXYCjgbOGc6QZC/g/cAzqup781tNSdJi0HSlUVUbk7wA+DiwBDitqi5Oclw//WTgH4DdgH9NArCxqlYtTLUlSZPQ2j1FVZ0LnDtt3MmD188Fnjt/VZMkLTY+ES5JamZoSJKaGRqSpGaGhiSpmaEhSWpmaEiSmhkakqRmhoYkqZmhIUlqZmhIkpoZGpKkZoaGJKmZoSFJamZoSJKaGRqSpGaGhiSpmaEhSWpmaEiSmhkakqRmhoYkqZmhIUlqZmhIkpoZGpKkZoaGJKmZoSFJamZoSJKaGRqSpGaGhiSpmaEhSWpmaEiSmhkakqRmhoYkqZmhIUlqZmhIkpo1h0aSw5NckmRNkhNGTN8/yVeS/GeSl8xvNSVJi8HSlpmSLAHeDBwGrAPOT3JOVX17MNvPgb8BjprvSkqSFofWK42DgTVVdWlVXQ+cCRw5nKGqrqqq84Eb5rmOkqRFojU09gTWDobX9eM2W5Jjk6xOsnr9+vVbsghJ0oS0hkZGjKstKbCqTqmqVVW1avny5VuyCEnShLSGxjpg5WB4BXD5/FdHkrSYtYbG+cB+SfZNshNwNHDOwlVLkrQYNX17qqo2JnkB8HFgCXBaVV2c5Lh++slJ7gasBu4I3JTkeOCAqrpuYaouSRq3ptAAqKpzgXOnjTt58PoKum4rSdI2yifCJUnNDA1JUjNDQ5LUzNCQJDUzNCRJzQwNSVIzQ0OS1MzQkCQ1MzQkSc0MDUlSM0NDktTM0JAkNTM0JEnNDA1JUjNDQ5LUzNCQJDUzNCRJzQwNSVIzQ0OS1MzQkCQ1MzQkSc0MDUlSM0NDktTM0JAkNTM0JEnNDA1JUjNDQ5LUzNCQJDUzNCRJzQwNSVIzQ0OS1MzQkCQ1MzQkSc0MDUlSM0NDktSsOTSSHJ7kkiRrkpwwYnqS/J9++jeSHDS/VZUkTVpTaCRZArwZOAI4ADgmyQHTZjsC2K//OxZ4yzzWU5K0CCxtnO9gYE1VXQqQ5EzgSODbg3mOBN5VVQWcl+ROSe5eVT+d1xpr4vY54SMLuvzLXvfYBV2+pC3XGhp7AmsHw+uAQxrm2RPYJDSSHEt3JQKwIcklzbWdX7sDV7fOnNdPptxJlr0NlLvZZW8D5U6y7O2t3EmWvfcEygTaQyMjxtUWzENVnQKc0ljugkmyuqpWbS/lTrJs13n7KHt7K3fSZU9K643wdcDKwfAK4PItmEeSdBvWGhrnA/sl2TfJTsDRwDnT5jkHeGb/LaoHAb/wfoYkbVuauqeqamOSFwAfB5YAp1XVxUmO66efDJwLPAZYA/waePbCVHneTKqLbJJdc67ztl/uJMve3sqddNkTke7LTpIkzc0nwiVJzQwNSVKz7S405vo5lAUs97QkVyX51rjK7MtdmeTfk3wnycVJXjTm8u+U5H1JvtvX4cFjKve/9ev7rSRnJLn9mMq9V5KLBn/XJTl+TGVfluSbfbmrx1HmoOwlSf4jyYfHWObtk3wtydf7ff2qMZb9or5tXTyu/btYbFf3NPqfQ/kecBjdV4TPB46pqm/P+sb5KfthwAa6p+bvu9DlDcq9O3D3qrowya7ABcBR41jnvvx3Al+oqrf137zbpaquXeAy9wS+CBxQVb9JchZwblW9YyHLHVGPJcBPgEOq6kdjKO8yYFVVjf1hsyR/C6wC7lhVjxtTmQGWVdWGJDvS7fMXVdV5C1zufYEz6X4p43rgY8Dzqur7C1nuYrG9XWnc/HMoVXU93Y4/chwFV9XngZ+Po6xp5f60qi7sX/8S+A7dk/oLLskdgYcBp/blX7/QgTGwFNg5yVJgFybzzNAjgR+MIzAmKckK4LHA28ZZbnU29IM79n/j+BR8b+C8qvp1VW0EPgc8cQzlLgrbW2jM9FMn24Uk+wAPAL46piJ/D1gPvL3vunhbkmULXWhV/QT4J+DHdD9j84uq+sRClzvC0cAZYyyvgE8kuaD/uZ5xOQl4KXDTGMsEbu4Wuwi4CvhkVY2jbX8LeFiS3ZLsQveowco53rPN2N5Co+mnTrZFSe4AnA0cX1XXjanYpcBBwFuq6gHAr4AFv4+U5M50V5D7AnsAy5I8faHLnVaHnYAnAO8dY7EPqaqD6H5x+vl9l+iCSvI44KqqumChyxqlqm6sqgPpfoHi4L7raKHL/A7weuCTdF1TXwc2LnS5i8X2Fhrb5U+d9P29ZwPvqar3j7HodcC6wae/99GFyEJ7FPDDqlpfVTcA7wf+aAzlDh0BXFhVV46rwKq6vP9/FfABuu7YhfYQ4An9/ZQzgUckefcYyt1E3+35WeDwMZV3alUdVFUPo+t23i7uZ8D2FxotP4eyTelvFp4KfKeq3jjOsqvqCmBtknv1ox7Jpj+nv1B+DDwoyS79+j+S7l7OOB3DGLumkizrv+hA3wX4p3TdKAuqql5eVSuqah+64+kzVTWWq7oky5PcqX+9M92Hhe+Oqey79v/3Ap7EeLshJ6r1V263CTP9HMo4yk5yBnAosHuSdcArq+rUMRT9EOAZwDf7vl+Av6uqc8dQNsALgff0IX0pY/h5mar6apL3ARfSdRv8B2P8uYe+n/sw4L+Oq0zgd4EPdBnJUuD0qvrYGMufhLsD7+y/pbYDcFZVjesrv2cn2Q24AXh+VV0zpnInbrv6yq0kaetsb91TkqStYGhIkpoZGpKkZoaGJKmZoSFJamZoSJKaGRqSpGb/HybkIgwMXH8HAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "indices = np.argsort(imp_vals)[::-1]\n",
    "plt.figure()\n",
    "plt.title(\"Random Forest feature importance via permutation importance\")\n",
    "plt.bar(range(X.shape[1]), imp_vals[indices])\n",
    "plt.xticks(range(X.shape[1]), indices)\n",
    "plt.xlim([-1, X.shape[1]])\n",
    "plt.ylim([0, 0.5])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, also here, features 1, 0, and 2 are predicted to be the most important ones, which is consistent with the feature importance values that we computed via the *mean impurity decrease* method earlier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Note that in the context of random forests, the feature importance via permutation importance is typically computed using the out-of-bag samples of a random forest, whereas in this implementation, an independent dataset is used.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Previously, it was mentioned that the permutation is repeated multiple times if `num_rounds > 1`. In this case, the second array returned by the `feature_importance_permutation` contains the importance values for these individual runs (the array has shape [num_features, num_rounds), which we can use to compute some sort of variability between these runs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdUAAAEICAYAAAAJGW4GAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAeo0lEQVR4nO3de7wdZX3v8c/PhCgEEIUokotBxQtVuZgCrRZjFQVBsbanBbwcOaU5tFCl1aNoq6Klp9bjUWuLcBBQKQaKKG2qEbC1wQugCZYqAdEYAonhEjDIxQtGfuePeTaZLNbee+3k2WutsD/v12u9sub6PDPzzHxnnjVJIjORJEnb7jGDroAkSY8WhqokSZUYqpIkVWKoSpJUiaEqSVIlhqokSZUMVahGxGkRccGg6zEMovHJiNgYEd8adH0mKiLuj4inDboe26OI+K2IuGnQ9RhGEfG6iLhiEtbrPh9yEbEmIl62lct+KiJOr12nbsYN1bIhPysXydtL5XbuR+UmS0QsjIiHyjaNfP61j+XPj4iMiOljzPYi4DBgTmYetI3lvSkivr4t65iozNw5M1f3s8zRlH39jEHXo1eZ+bXMfNag61HbRNtht/MkMz+TmS+vXbdh2ufl+rRu0PXop20JzGHT65PqqzJzZ2B/4ADgnZNWo/5ZXy78I59XTXQFETFtMipWPBVYk5kPTGIZPRkn/IfW9lrvYeH+6z/3+aNAZo75AdYAL2sNfxD4Ymv4VOCHwH3ADcDvtKa9Cfg68CFgI3AzcERr+t7AlWXZLwP/AFzQmv5qYCVwD7AMeE5Hvf4X8B3gAeBc4MnAl8r6/g14wijbtBBYN8q055Sy7illv7o17VPAmcDSUubLgL2AzwEbyva9uTX/QcAK4F7gDuDDZfytQAL3l89vdNThD4GfA78q099Xxh8FXFfqdhXw/PGOQ9me9rruKeOXASd0HqvWcAInAT8Abh6v/C77MYFntPbbx8uxuR/4BrAn8NHSLr4HHNBxbN9ZtmMj8Engca3pfwSsAn4MLAH2Gq3ewFfLuAdK2X8APAH4QjlmG8v3Oa11LAP+qtTzPuAKYI/W9BeV7b8HWAu8qYx/LE1bv7Uc77OAHbvsm8eWZZ/bGjcL+BnwJDra52jHdpT9fhpwCfBPZf5vA/u1po/VXkeWvYCmzZ5Q9sXpZXvvB/4V2B34TJlnOTC/LD+/7OvpHfvyBEZvh0cC/1nWtRY4rbXsI84THtlOf7PU4Sflz9/s9TiOdU1gAteX1nYvAtYDtwFv7TjeHy3T1pfvj22XC7wDuB34bGkHD7W2ey+aa8nVNO3mNppr5YyOdn8iTbvfCJwBRMc5cyOb29CB47WHjv2zdyn7MWX4HODO1vQLgFPGyZI9aM61e2jO3a/RPNj9Y9nen5XtfXuZ/w3ALcDdwF/QkUXjlHUATdu/j+ZcuAg4vTW967WM5ly7pGNdfwd8rJdyM3NioQrMAb4L/F1r+n8rB+YxNBesB4CntC7UvywHdBrwx6VRRZl+NfDh0ugOLTvggjLtmWVdhwE7AG+nuZDOaNXrGpqGPhu4s+zEA8r6vgK8t5cTqDV+h1LGu4AZwG+XOj2rFQ4/AV5Ytncn4FrgPWX+pwGrgVe0tu8N5fvOwCGjXXy61OVNbHnxOLBs48FlX/73sg8e2+Nx+HrH+pcxfqh+GXgisON45Xepf2eo3gW8AHhcOTY3A28s6zod+I+ONnc9MLeU/w3KCVGOyV2lPo8F/h746mj17qxLGd4d+N1y/HahuZD9c8e++SFNG9yxDH+gTJtX2sSxNO1ld2D/Mu2jNCH/xLLefwX+ZpT9cx7w163hk4DLRrnAj3psu6z3NJpz7vdK/d5W9vUOZfmx2uvIsq8p845s+yrg6cDjaS7I36e5oZwOnA98crR2Taud0b0dLgSeV8p7Ps3NyGvGWN/D6yj7eSPNxXd6OSYbgd3HO47jXROYwPWlVc8LgZllezaw+br5/rKuJ9HcPF0F/FWr3E3A35b17thZlzLfC4BDynbOpwnIU1rTkyawdqNpoxuAw1vt50fArwMBPIOmJ2zM9tBlH90KvKB8v6nM+5zWtAO6Ldda/m9objR3KJ/fYnMWrGHLh7d9aQL20LJfPlz207ihWrblFuDPSjm/R9OuR64ho17Lyn75KbBrmXcazU3MIeOV+3D5PVRwTdm4+8qB+3dgtzHmvw44unUCrGpN26msY89y4DcBM1vTF7M5VN8NXNya9pjSMBa26vW61vTPAWe2hv+U1oWyywn0EM1dysjn98tBvp1yN1bmvZBy90wTDue3ph0M3Nqx7ney+SLzVeB9dNwds3WheiblRGyNuwl4cY/HYWtC9be3ofzOUP1Ex7G5sTX8PMqTS+vYntgafiXww/L9XOCDrWk705ww87vVu7Muo9R1f2Bjx775y9bwn7A58N4JXNplHUETdk9vjfsNylN+l/lfBqxuDX8DeGOrfXbtSek8tl2mnQZc03He3EbTtsdrr6fRukFp7Yu/aA3/X+BLreFXAdeN1q4ZJ1S71P+jwEfGWN/D66AJ0291LH81m3sORj2OXcrdYp8zgetLq57Pbk3/IHBu+f5D4JWtaa+g+WlnpNwH2bInZszjX+Y5pd0OS/kvag1fDJxavl8OvKXLOsZsD13m/0fgz2mu3zeVbTyRjqfYMer8fuBf6HIu8shQfQ9wUWt4ZtlPvYTqobQe3sq4q9gcqmNey2h6V0fOxcMo155eP73+pvqazNylHOxn0zzGAxARb4yI6yLinoi4B3huezpNSAGQmT8tX3emufPemFv+ZnhL6/te7eHMfIime2h2a547Wt9/1mV4rBeq1mfmbq3PxaXMtaWsdp3aZa5tfX8qsNfItpftfxfN3S003bjPBL4XEcsj4qgx6jOepwJv7ShrbqlzL8dha3Ru66jl92Cix6pd9i2tcjrbxf003UOjHaNHiIidIuL/RcQtEXEvzc3Pbh2/kd/e+v7TVv3m0lwkO82i9Fy09s9lZXw3XwF2jIiDI+KpNMF+6Sj1neixfXj7S1teR7PfxmuvWyzbsi3n2ZjK9v9HRGyIiJ/QXKR7bbdbtIWi83wd7Tj2YlLabMc0gA2Z+fOxKhIRz4yIL5SXRe8F/jeP3E8TbbO9tIe2K2ky4FCac2YZ8OLy+VrHdbOb/0PT63FFRKyOiFPHmHcvtmzHD9Cc573YC/hRllQs2vt/vGvZYppeD4DjynDPJvRXajLzSpqnjg8BlIvBJ4CTabpcdqPptoseVncb8ISImNkaN6/1fT3NxlPKCpoN/9FE6jxB64G5EdHeL/M6ymwfqLU0TyLtcN4lM18JkJk/yMxjabp9/ha4pGxvex29WkvTXdgua6fMvLCH49CtvAdoQmDEnl3m6dzWruVvxbb0Ym7r+zyaYwOPbBczabpgRztG3bwVeBZwcGbuSnORgN7a7VqartBOd9FcaH+ttX8en80Lfo9QLkAX05y8xwFfyMz7OufbynPs4X1X2vIcmv02ZnsdqdoY6x3PyA3yaO2q27oX03SZz83Mx9N0D47Vbtu2aAtF5/naTz212Y5p8Mjt7LbdZ9K8e7BPabPvorf2CqO32V7aQ9uVND0eC8v3r9P8FPbiMjymzLwvM9+amU+j6eH484h46cjkjtlvY8t2vBPNed6L24DZJTNGtLNlvGvZZ4GFETEH+B0mM1SLjwKHRcT+NI/kSdN/T0QcT3MXPa7MvIXmJZ73RcSMiHgRzY4ecTFwZES8NCJ2oLkQ/oLmMX6yfJPmwvD2iNghIhaWOl00yvzfAu6NiHdExI4RMS0inhsRvw4QEa+PiFnlAnpPWeZXNPvrIZrfMHr1CeDEcmcfETEzIo6MiF0Y/zjcAcyJiBmtcdcBry1Pbc+geare2vInw0kRMScinkhzAfmnMn4xcHxE7B8Rj6W5Y/9mZq4ZY113sOW+3oUmAO8p63/vBOr1GeBlEfH7ETE9InaPiP3LMf4E8JGIeBJARMyOiFeMsa7FNL+Rvo7RT9ytOcdeEBGvLW+SnkJz3lzDOO11W2XmBppAe31Z9/9gy4t5t3a4C/DjzPx5RBxEc4MxYrzzZCnwzIg4rhyLP6D5Le4LNbZnK7y7nE+/BhzP5jZ7IfCXETErIvag6doc6+/j3wHsHhGPb43bheZlrvsj4tk076f06hzgbRHxgnLuPqPcrE2oPWTmD2jOm9fT/Eww8gLm79JDqEbEUaXsKNvyq/IZ2eb2cb4EOCoiXlTay/vpPa+upvlp8c2lXbyW5kWvEWNey0o7XkbzguTNmXljj+XCBCr5sFLg+cC7M/MGmt9YrqbZKc+j+W2oV8fR9Ov/mObCdn6rnJtoDt7f0zwFvIrmr/Y8ONE696qs+9XAEaXMj9P0rX9vlPl/Veq1P83LIHfRNOCRk+FwYGVE3E/zBtkxmfnz0g3+18A3SvfDIT3UbQXNC1//QPMyxiqa35fo4Th8heZN5tsj4q4y7iM0v1HcAXyaJiy2qvxJspjmbc3V5XN6qce/0/ze/jmaO9KnA8eMs67TgE+Xff37NDeGO9Icr2touml7kpm30vzG+1aadnsdsF+Z/A6a/XJNNF10/0bzRDzaukZu4vaieau02zxbc479C01Yb6T53fG1mfnLHtprDX9E89bs3cCvseVNcLd2+CfA+yPiPpqwuXhk5vHOk8y8m+YtzreW8t4OHJWZdzEYV9Ic/38HPpSZI/9Ixek0DxDfoXnR89tlXFflenMhsLps9140L5wdR/NuyyfYHNjjyszP0uzHxWX5fwaeuJXt4Urg7nIejAwHzRvcRMSXIuJdoyy7D805cT9Ne/54Zi4r0/6G5sbjnoh4W2aupHl5bzHNeb6R5mcMSjmvi4iVo2zvg8Braa5PG2nOhc+3pvdyLVtM897Dwze7EfGuiOh6nraNvHklDY2IWEPzcsu/Dbou25uIOI3mRZDXD7ouU0VEzKe8YZ2ZmwZcHQ3YUP0zhZIkbc8MVUmSKrH7V5KkSnxSlSSpEv/x5pY99tgj58+fP+hqSNJ25dprr70rM0f7h06mFEO1Zf78+axYsWLQ1ZCk7UpEdP7LVlOW3b+SJFViqEqSVImhKklSJYaqJEmVGKqSJFViqEqSVImhKklSJYaqJEmVGKpDYOHChSxcuHDQ1ZAkbSNDVZKkSgxVSZIqMVQlSarEUJUkqRL/l5pJMv/UL/Y87+2r757wMms+cOSE69Rp5OWoZcuWbfO6JEk+qUqSVI2hKklSJYaqBsK/myvp0chQlSSpEl9UepTZHl6QkqRHK0N1COx53AcGXQVJUgV2/0qSVIlPqqpiIl3IYNezpEcnn1QlSarEJ9UpzN9yJakun1QlSapk6EM1Ig6PiJsiYlVEnNpl+usi4jvlc1VE7NeatiYivhsR10XEiv7WXJI01Qx1929ETAPOAA4D1gHLI2JJZt7Qmu1m4MWZuTEijgDOBg5uTX9JZt7Vt0pLkqasoQ5V4CBgVWauBoiIi4CjgYdDNTOvas1/DTCnrzXUVvH3XEmPRsPe/TsbWNsaXlfGjeYPgS+1hhO4IiKujYhFk1A/SZIeNuxPqtFlXHadMeIlNKH6otboF2bm+oh4EvDliPheZn61Y7lFwCKAefPm1am1JGlKGvYn1XXA3NbwHGB950wR8XzgHODozLx7ZHxmri9/3glcStOdvIXMPDszF2TmglmzZlWuviRpKhn2UF0O7BMRe0fEDOAYYEl7hoiYB3weeENmfr81fmZE7DLyHXg5cH3fai5JmnKGuvs3MzdFxMnA5cA04LzMXBkRJ5bpZwHvAXYHPh4RAJsycwHwZODSMm46sDgzLxvAZkiSpoihDlWAzFwKLO0Yd1br+wnACV2WWw3s1zlekqTJMuzdv5IkbTcMVUmSKjFUJUmqxFCVJKkSQ1WSpEoMVUmSKjFUJUmqxFCVJKkSQ1WSpEoMVUmSKjFUJUmqxFCVJKkSQ1WSpEoMVUmSKjFUJUmqxFCVJKkSQ1WSpEoMVUmSKjFUJUmqxFCVJKkSQ1WSpEoMVUmSKjFUJUmqxFCVJKkSQ1WSpEoMVUmSKjFUJUmqZOhDNSIOj4ibImJVRJzaZfrrIuI75XNVROzX67KSJNU01KEaEdOAM4AjgH2BYyNi347ZbgZenJnPB/4KOHsCy0qSVM1QhypwELAqM1dn5oPARcDR7Rky86rM3FgGrwHm9LqsJEk1DXuozgbWtobXlXGj+UPgSxNZNiIWRcSKiFixYcOGbayuJGkqG/ZQjS7jsuuMES+hCdV3TGTZzDw7Mxdk5oJZs2ZtdUUlSZo+6AqMYx0wtzU8B1jfOVNEPB84BzgiM++eyLKSJNUy7E+qy4F9ImLviJgBHAMsac8QEfOAzwNvyMzvT2RZSZJqGuon1czcFBEnA5cD04DzMnNlRJxYpp8FvAfYHfh4RABsKt25XZcdyIZIkqaEoQ5VgMxcCiztGHdW6/sJwAm9LitJ0mQZ9u5fSZK2G4aqJEmVGKqSJFViqEqSVImhKklSJYaqJEmVGKqSJFViqEqSVImhKklSJYaqJEmVGKqSJFViqEqSVImhKklSJYaqJEmVGKqSJFViqEqSVImhKklSJYaqJEmVGKqSJFViqEqSVImhKklSJYaqJEmVGKqSJFViqEqSVImhKklSJYaqJEmVDH2oRsThEXFTRKyKiFO7TH92RFwdEb+IiLd1TFsTEd+NiOsiYkX/ai1JmoqmD7oCY4mIacAZwGHAOmB5RCzJzBtas/0YeDPwmlFW85LMvGtSKypJEsP/pHoQsCozV2fmg8BFwNHtGTLzzsxcDvxyEBWUJGnEsIfqbGBta3hdGderBK6IiGsjYlG3GSJiUUSsiIgVGzZs2IaqSpKmumEP1egyLiew/Asz80DgCOCkiDj0ESvLPDszF2TmglmzZm1tPSVJGvpQXQfMbQ3PAdb3unBmri9/3glcStOdLEnSpBj2UF0O7BMRe0fEDOAYYEkvC0bEzIjYZeQ78HLg+kmrqSRpyhvqt38zc1NEnAxcDkwDzsvMlRFxYpl+VkTsCawAdgUeiohTgH2BPYBLIwKa7VycmZcNYDMkSVPEUIcqQGYuBZZ2jDur9f12mm7hTvcC+01u7SRJ2mzYu38lSdpuGKqSJFViqEqSVImhKklSJYaqJEmVGKqSJFViqEqSVImhKklSJYaqJEmVGKqSJFViqEqSVImhKklSJYaqJEmVGKqSJFViqEqSVImhKklSJYaqJEmVGKqSJFViqEqSVImhKklSJYaqJEmVGKqSJFViqEqSVImhKklSJYaqJEmVGKqSJFUy9KEaEYdHxE0RsSoiTu0y/dkRcXVE/CIi3jaRZSVJqmmoQzUipgFnAEcA+wLHRsS+HbP9GHgz8KGtWFaSpGqGOlSBg4BVmbk6Mx8ELgKObs+QmXdm5nLglxNdVpKkmoY9VGcDa1vD68q4astGxKKIWBERKzZs2LDVFZUkadhDNbqMy5rLZubZmbkgMxfMmjVrQpWTJKlt2EN1HTC3NTwHWN+HZSVJmrBhD9XlwD4RsXdEzACOAZb0YVlJkiZs+qArMJbM3BQRJwOXA9OA8zJzZUScWKafFRF7AiuAXYGHIuIUYN/MvLfbsgPZEEnSlDDUoQqQmUuBpR3jzmp9v52ma7enZSVJmizD3v0rSdJ2w1CVJKkSQ1WSpEoMVUmSKjFUJUmqxFCVJKkSQ1WSpEoMVUmSKjFUJUmqxFCVJKkSQ1WSpEoMVUmSKjFUJUmqxFCVJKkSQ1WSpEoMVUmSKjFUJUmqxFCVJKkSQ1WSpEoMVUmSKjFUJUmqxFCVJKkSQ1WSpEoMVUmSKjFUJUmqxFCVJKmSoQ/ViDg8Im6KiFURcWqX6RERHyvTvxMRB7amrYmI70bEdRGxor81lyRNNdMHXYGxRMQ04AzgMGAdsDwilmTmDa3ZjgD2KZ+DgTPLnyNekpl39anKkqQpbNifVA8CVmXm6sx8ELgIOLpjnqOB87NxDbBbRDyl3xWVJGnYQ3U2sLY1vK6M63WeBK6IiGsjYlG3AiJiUUSsiIgVGzZsqFRtSdJUNOyhGl3G5QTmeWFmHkjTRXxSRBz6iBkzz87MBZm5YNasWdtWW0nSlDbsoboOmNsangOs73WezBz5807gUpruZEmSJsWwh+pyYJ+I2DsiZgDHAEs65lkCvLG8BXwI8JPMvC0iZkbELgARMRN4OXB9PysvSZpahvrt38zcFBEnA5cD04DzMnNlRJxYpp8FLAVeCawCfgocXxZ/MnBpRECznYsz87I+b4IkaQoZ6lAFyMylNMHZHndW63sCJ3VZbjWw36RXUJKkYti7fyVJ2m4YqpIkVWKoSpJUiaEqSVIlhqokSZUYqpIkVWKoSpJUiaEqSVIlhqokSZUYqpIkVWKoSpJUiaEqSVIlhqokSZUYqpIkVWKoSpJUiaEqSVIlhqokSZUYqpIkVWKoSpJUiaEqSVIlhqokSZUYqpIkVWKoSpJUiaEqSVIlhqokSZUYqpIkVTL0oRoRh0fETRGxKiJO7TI9IuJjZfp3IuLAXpeVJKmmoQ7ViJgGnAEcAewLHBsR+3bMdgSwT/ksAs6cwLKSJFUzfdAVGMdBwKrMXA0QERcBRwM3tOY5Gjg/MxO4JiJ2i4inAPN7WFaPAvNP/WLP896+uOmw2PO4D/S8zJoPHDnhOnVauHAhAMuWLdvmdW0vZQ9ym6VBGfZQnQ2sbQ2vAw7uYZ7ZPS5LRCyiecJl3rx5217josaF2HInoewB1dNg2XaDunmaSLlbY1DljlW2tt6wh2p0GZc9ztPLsmTm2cDZAAsWLHjEdGl7N6hAr13uoG6eJlJuzafziQaePQPDYdhDdR0wtzU8B1jf4zwzelhWkqoZZKAZpsNhqF9UApYD+0TE3hExAzgGWNIxzxLgjeUt4EOAn2TmbT0uK0lSNUP9pJqZmyLiZOByYBpwXmaujIgTy/SzgKXAK4FVwE+B48dadgCbIUmaIqJ5aVbQ/Ka6YsWKQVdDkrYrEXFtZi4YdD2GwbB3/0qStN0wVCVJqsRQlSSpEkNVkqRKDFVJkioxVCVJqsS/UtMSERuAWwZU/B7AXVOo3EGWPdXKHWTZbvPUKPupmTlrAOUOHUN1SETEikH8Pa9BlTvIsqdauYMs222eOmWrYfevJEmVGKqSJFViqA6Ps6dYuYMse6qVO8iy3eapU7bwN1VJkqrxSVWSpEoMVUmSKjFUBywizouIOyPi+gGUfXhE3BQRqyLi1D6VOTci/iMiboyIlRHxln6U2yp/t4i4JCK+V+rwG30qd01EfDcirouIvv3/ghHxrFLmyOfeiDilT2X/WTnG10fEhRHxuH6UW8qeFhH/GRFf6FeZpdy3lO1d2cf9/LiI+FZE/Fcp9339KFfd+ZvqgEXEocD9wPmZ+dw+ljsN+D5wGLAOWA4cm5k3THK5TwGekpnfjohdgGuB10x2ua3yPw18LTPPiYgZwE6ZeU8fyl0DLMjMQf2jACPH/EfAwZk5qf/ISUTMBr4O7JuZP4uIi4GlmfmpySy3Vf6fAwuAXTPzqD6V+VzgIuAg4EHgMuCPM/MHk1xuADMz8/6I2IFmv78lM6+ZzHLVnU+qA5aZXwV+PICiDwJWZebqzHyQ5mJw9GQXmpm3Zea3y/f7gBuB2ZNdLkBE7AocCpxbyn+wH4E6RF4K/HCyA7VlOrBjREwHdgLW96PQiJgDHAmc04/yWp4DXJOZP83MTcCVwO9MdqHZuL8M7lA+Pi0NiKE6dc0G1raG19GncBsREfOBA4Bv9qnIpwEbgE+WrsFzImJmn8pO4IqIuDYiFvWpzE7HABf2o6DM/BHwIeBW4DbgJ5l5RT/KBj4KvB14qE/ljbgeODQido+InYBXAnP7UXDp7r4OuBP4cmb265xSB0N16oou4/p2dxsROwOfA07JzHv7VOx04EDgzMw8AHgA6MtvycALM/NA4AjgpNLt3zelq/vVwGf7VN4TaHo+9gb2AmZGxOv7UO5RwJ2Zee1kl9UpM28E/hb4Mk3X738Bm/pU9q8yc39gDnBQ6YrWABiqU9c6tryLnkP/uud2oAnUz2Tm5/tRZrEOWNe6i7+EJmQnXWauL3/eCVxK0/3eT0cA387MO/pU3suAmzNzQ2b+Evg88Jt9KPeFwKvLb9gXAb8dERf0oVwAMvPczDwwMw+l+VlnUn9P7VL+PcAy4PB+lqvNDNWpazmwT0TsXZ5ijgGWTHah5aWKc4EbM/PDk11eW2beDqyNiGeVUS8FJv0FqYiYWV7KonQ3v5ymq7CfjqVPXb/FrcAhEbFTOeYvpfn9fFJl5jszc05mzqdp01/JzEl/Qh4REU8qf84DXksf9nlEzIqI3cr3HWluaL432eWqu+mDrsBUFxEXAguBPSJiHfDezDx3ssvNzE0RcTJwOTANOC8zV052uTRPEm8Avlt+AwJ4V2Yu7UPZAH8KfKbcSKwGju9DmU8GLm2yhenA4sy8rA/lAlB+3zsM+J/9KjMzvxkRlwDfpukC/U+mxj+h97mI2B34JXBSZm7sQ5lPAT5d3u5+DHBxZvb1rxJpM/9KjSRJldj9K0lSJYaqJEmVGKqSJFViqEqSVImhKklSJYaqJEmVGKqSJFXy/wHPrHFC7Qg/hAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "imp_vals, imp_all = feature_importance_permutation(\n",
    "    predict_method=forest.predict, \n",
    "    X=X_test,\n",
    "    y=y_test,\n",
    "    metric='accuracy',\n",
    "    num_rounds=10,\n",
    "    seed=1)\n",
    "\n",
    "\n",
    "std = np.std(imp_all, axis=1)\n",
    "indices = np.argsort(imp_vals)[::-1]\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"Random Forest feature importance via permutation importance w. std. dev.\")\n",
    "plt.bar(range(X.shape[1]), imp_vals[indices],\n",
    "        yerr=std[indices])\n",
    "plt.xticks(range(X.shape[1]), indices)\n",
    "plt.xlim([-1, X.shape[1]])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It shall be noted that the feature importance values do not sum up to one, since they are not normalized (you can normalize them if you'd like, by dividing these by the sum of importance values). Here, the main point is to look at the importance values relative to each other and not to over-interpret the absolute values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Support Vector Machines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the permutation importance approach yields results that are generally consistent with the *mean impurity decrease* feature importance values from a random forest, it's a method that is model-agnostic and can be used with any kind of classifier or regressor. The example below applies the `feature_importance_permutation` function to a support vector machine:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy 94.87142857142857\n",
      "Test accuracy 94.89999999999999\n"
     ]
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "\n",
    "\n",
    "svm = SVC(C=1.0, kernel='rbf')\n",
    "svm.fit(X_train, y_train)\n",
    "\n",
    "print('Training accuracy', np.mean(svm.predict(X_train) == y_train)*100)\n",
    "print('Test accuracy', np.mean(svm.predict(X_test) == y_test)*100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAbyUlEQVR4nO3debhdVZ3m8e/LDZMhECFhykBQ06XRBiudBgQKgggSsCpYE4NCQUlFniaNVGtZkfZRHKrBKqta7EJTPIDgEFKKxEprGFXksRHIDTIFCIYQyCVgLmEeNAR+/cdaF3YO5967T3LPucnK+3me+9yz99p7r7WH85511pkUEZiZWbm2Ge4GmJlZeznozcwK56A3Myucg97MrHAOejOzwjnozcwK56BvkaRDJP1G0guSjh/u9rRC0rmSLhnudmypJF0j6a+Gux2bI0lLJU1vw3Z9zIeAttT30Us6FPhH4N3Aq8D9wDnACOBGYM+IeL5hnV8DlwI/Bh4Gfh0RUyvlY4DVwOqImNRPvT8FFkbEhUOwDyuBMyLixk3d1pZG0nnAOyLio8Pdlq1dq9ehpMuBnoj4bDvbtbmRdBPw3YjY4jpLW2SPXtLOpLD+P8CuwDjgC8DvI+JXQA/wZw3rvAeYAlxZmT0yz+9zMukBYCD7AEs3aQeGiKQRw92GjbGltntz4GPXeUq2yKx8XURscX/ANOCZAcrPBX7WMO8fgavz7UlAAJ8F/qmyTDfwP4GV/Wz3IeA14GXgBWB7YBfSs4THgceALwNdefm3Az8D1gJPAt8DRuey7zRs69PAdFJPqVrnSuAD+fZ5wFXAd4HngDMGqr9J+88j9Uiqx+B0YBXwNHAm8F+Bu4FngH+trHsa8P9ID67PAg8AR1bK9wYWAk8By4G/aai32u7ZwDrglbzvd+XlTic9M3seWAF8vLKN6aQH8E8Ca/L+nl4p3xH4Z+CR3L5fAjvmsoOAW/I+3QVM7+f4zAGuaph3IfD1fPsmUs93wHPbz7YDODvv15PAPwHbVMr/Ou/708B1wD4N654F/IbUEek7Fp+uHIvjgWOBB/M5OLey/uXAlxuPZX/XYZ7/A+CJfCxvBt6d58/K521dXv7/NrlOtwe+Rn52nG9vX+c8Njlu1WN+Guka/N/5XK4ADs7zV+Xt/VXDfs8FbiBdU79oOK4HA4vzPi4GDm6o9x9yfS/n8/sq8Lu83/9auT5Wka7rJcAfNVz33we+netfCkyrlE8ArgZ6SddR9f7W7/WwUZk5VOHbyT9g53xgrgBmAG9tKJ+QL8aJeXqbfHEdHxuG3KR8krqAdwHLgA/QT9A3XtB5+kfAvwEjgd2B28kBBbwDOCpf+GNJd5ivDbCt6Qwe9K+Q7tTbkMKt3/qbtP083hz0c4EdgKPzRfyjvJ1xpDvO4ZU72Xrgb4FtgRNId5Bdc/kvgG/kbb03X7xHDtDu19tSad9xpAAVcDjwEjC1cmzWA1/M9R+by9+ayy8i3TnH5fN5cD7u40jXyrG57qPy9Ngmx2efvM2d83QXKYgOahI6A57bJtsO4OekZ6ATSYHct63jSQ+O7yINPX4WuKVh3RvyujtWjsXn8rH4m3y85wGjSMOZvwPeVgm8pkHf7DqsBM0o3gjtOytlG2yvyXX6ReBW0nU0lvQg+6U657HJcase89Pyuqfnc/Nl4NF87rcnXcPPAztV2vk8cFguvxD4ZS7blRSip+RjflKe3q1S76P5WI7IbX29LZX2fRTYLS/zSdKD4w6V6/53eR+7gPOBWyvX1l2kB62RpPvNoXWuh43KzOEK6039ywfhclKAryf1JveolN9I7tWQ7pBPAts2hFzfeP4HgQtIvfnaQQ/sAfye3HPM804Cft7PuseTXhfYlKC/uVLWav3n8eagH1cpXwucUJn+IXBO5U62mvy6Tp53O+mOMoHU2xlVKTsfuLxZuxvbMsCx/hHwicqxeRkYUSlfQ+qtb5PL9m+yjb8HvtMw7zoqPb+Gsl8Cp1aum4cqZTfRcEfv79w2KQ/gmMr0fwN+mm9fA3ysUrYNKfz2qaz7/obr5GXeeOY4Ki9zYGWZJbzRsbmcFoO+oe2j8/Z3aba9JtfpQ8CxlbIPku9TA53Hfup+/Zjna/A3lbL/nNtVvd+vBd5baef8StlOpOt0Aum6vb2hrl8Bp1Xq/WJ/bRngWD3ddx2SrvEbK2VTgJfz7feRHpxHNNnGgNfDxvxtseNOEXF/RJwWEeOB95CGDr5WWeQK4NR8+xRgXkS80mRT3yZdQCeRhhZasQ/pkf5xSc9IeobUu94dQNLukuZLekzSc3n7Y1qso9GquvXX9NvK7ZebTO9UmX4s8pWXPUI67nsDT8WGL34/QupNN2t3U5JmSLpV0lN5X45lw+O1NiLWV6Zfyu0bQ+oRPdRks/sAf9F3fPJ2DwX26qcZ80jXAqTXbOb109aNObfVY9B37PraeGGlfU+RntUMdPzWRsSr+fbL+f9A5642SV2SLpD0UN63lbmo7rW7N2n/+lT3Ffo/j3U07iMRMdB+v37cIuIF0rHtu2arbexrZ6vX7Ccl3S/p2XzudmHD4/RE5fZLwA75dZYJwCMNx6FPneuhJVts0FdFxAOkR+/qC6tXA+MkHQH8KSnQm/khachgRUQ0nvjBrCL1qMdExOj8t3NEvDuXn0/qcewXETuTnuap2vSG7b0IvKVvQlIX6alvVXWdweofauMkVds/kTfGYXeVNKqh7LF+2v2maUnbk87FV0k9tNHAIjY8Xv15kvQU+e1NylaRevSjK38jI+KCfrb1A2C6pPHAh+kn6Bn83DYzoXK779j1tfHjDW3cMSJuqSzfePxascF1BezZUN647ZOBmaRnt7uQnv3BG/s3WFtWk8KqT3VfO+31Yy5pJ9KQTd81u0/Dsq1es39Eesb4l6Shp9Gk4cw61+wqYGI/L67XuR5askUGvaR35kfS8Xl6AqkXdmvfMhHxIukFwG+RHjm7m20rL/d+0gubLYmIx4HrgX+WtLOkbSS9XdLheZFRpBdunpE0Dvi7hk38FnhbZfpB0iP+cZK2JY3Nbb8J9Q+13YGzJW0r6S9Iw2eLImIVaRz2fEk7SNoP+BjpBaz+/BaYVHk3w3akfe0F1kuaQRpzHVREvAZcBvyLpL1zj/R9+cHju8AfS/pgnr+DpL4gb7atXtJT9G8BD0fE/f1UO9i5bebvJL01X6+fAP49z58LfEbSuwEk7ZKP71C5EzhW0q6S9iS9Dbmq8TocRepArCU9QPyvQZZvdCXwWUlj81uWP0frz5aHyrGSDpW0HfAl4LZ8vS4C/pOkkyWNkHQCaWjlxwNsq9lxWk8egpH0OdLrh3XcTnr95wJJI/N1eUguG/LrYYsMetILLAcCt0l6kRTw95JeDKm6gvSo3V9vHoCI6I6IZk/76ziVFFL3kcbnruKNYYEvAFNJj/I/IT3LqDqfdId4RtKnIuJZ0tjtJaSexYuk1yA2tv6hdhswmdSD/gfgzyNibS47idTzWw0sAD4fETcMsK0f5P9rJd2Rh33OJr1L4WlSr3JhC237FHAP6d0TTwFfIb2rZRWpd3ou6Q65ihTKA13780i92f568zD4uW3mP0hj53fmdS4FiIgFub3z81DJvaQ3GQyV75Be+FtJ6hj8e0P5Btch6f7yCOkavI9KByq7FJiSl/9Rk/q+THoH292kc3JHnjcc5gGfJ10T/wX4CEC+bj9Eyoy1pHcwfSginhxgWxcCfy7paUlfJ73Wcw2pg/YI6VnloMM9uf5XgT8mvaj/KOl+fkIuG/LrYYv9wJR1lqTTSC9EHTrcbdkSSQpgckQsH+62bC221g92NbOl9ujNzKwmB72ZWeE8dGNmVjj36M3MCrdZfkHSmDFjYtKkScPdDDOzLcaSJUuejIjGz90Am2nQT5o0ie7upm97NzOzJiT1+4FPD92YmRXOQW9mVjgHvZlZ4Rz0ZmaFqxX0ko6RtEzScklzmpTPlHS3pDsldSv9nmutdc3MrL0GDfr8VbkXkb5UZwpwkqQpDYv9lPRl++8l/TLNJS2sa2ZmbVSnR38AsDwiVkTEOmA+6dsAXxcRL1R+kGIkb3xv86DrmplZe9UJ+nFs+NWbPTT5pRNJH5b0AOnrV/+6lXXNzKx96gR9s19LedMX5ETEgoh4J+m3M7/UyroAkmbl8f3u3t7eGs0aWtOnT2f69Okdr9fMrN3qBH0PG/4E2ngG+FmwiLgZeHv+ZZna60bExRExLSKmjR3b9FO8Zma2EeoE/WJgsqR9889xnUjDL/9Iekffb4lKmkr6xaO1ddY1M7P2GvS7biJivaTZpJ/N6gIui4ilks7M5XOBPwNOlfQK6VfYT8gvzjZdt037YmZmTdT6UrOIWET6Md3qvLmV218h/cZhrXXNzKxzNstvrxwqk+b8pPayT6xY2/I6Ky84ruU2mZl1mr8CwcyscA56M7PCOeiHmd+/b2btVvQYfSv2PPmC4W6CmVlbuEdvZlY4B72ZWeEc9GZmhXPQm5kVzkFvZlY4v+umDdr9iVzwp3LNrD736M3MCuegNzMrnIPezKxwDnozs8I56M3MCud33Qwzf8eOmbWbe/RmZoVz0JuZFc5Bb2ZWOAe9mVnhHPRmZoVz0JuZFc5Bb2ZWuFpBL+kYScskLZc0p0n5RyTdnf9ukbR/pWylpHsk3Smpeygbb5vGP0xutnUY9ANTkrqAi4CjgB5gsaSFEXFfZbGHgcMj4mlJM4CLgQMr5UdExJND2G4zM6upTo/+AGB5RKyIiHXAfGBmdYGIuCUins6TtwLjh7aZZma2seoE/ThgVWW6J8/rz8eAayrTAVwvaYmkWf2tJGmWpG5J3b29vTWaZWZmddT5rhs1mRdNF5SOIAX9oZXZh0TEakm7AzdIeiAibn7TBiMuJg35MG3atKbbt8G1+9et/MtWZlueOj36HmBCZXo8sLpxIUn7AZcAMyNibd/8iFid/68BFpCGgszMrEPqBP1iYLKkfSVtB5wILKwuIGkicDVwSkQ8WJk/UtKovtvA0cC9Q9V4MzMb3KBDNxGxXtJs4DqgC7gsIpZKOjOXzwU+B+wGfEMSwPqImAbsASzI80YA8yLi2rbsiZmZNVXr++gjYhGwqGHe3MrtM4Azmqy3Ati/cb6ZmXWOf3hkK+YfPTHbOvgrEMzMCuegNzMrnIPezKxwDnozs8I56M3MCuegNzMrnIPezKxwDnozs8I56M3MCuegNzMrnIPezKxwDnozs8I56M3MCuegNzMrnIPezKxwDnozs8I56M3MCuegNzMrnIPezKxwDnozs8I56M3MCuegNzMrnIPezKxwtYJe0jGSlklaLmlOk/KPSLo7/90iaf+665qZWXsNGvSSuoCLgBnAFOAkSVMaFnsYODwi9gO+BFzcwrpmZtZGdXr0BwDLI2JFRKwD5gMzqwtExC0R8XSevBUYX3ddMzNrrzpBPw5YVZnuyfP68zHgmlbXlTRLUrek7t7e3hrNMjOzOuoEvZrMi6YLSkeQgv7vW103Ii6OiGkRMW3s2LE1mmVmZnWMqLFMDzChMj0eWN24kKT9gEuAGRGxtpV1zcysfer06BcDkyXtK2k74ERgYXUBSROBq4FTIuLBVtY1M7P2GrRHHxHrJc0GrgO6gMsiYqmkM3P5XOBzwG7ANyQBrM/DME3XbdO+mJlZE3WGboiIRcCihnlzK7fPAM6ou66ZmXWOPxlrZlY4B72ZWeEc9GZmhXPQm5kVzkFvZlY4B72ZWeEc9GZmhXPQm5kVzkFvZlY4B72ZWeEc9GZmhXPQm5kVzkFvZlY4B72ZWeEc9GZmhXPQm5kVzkFvZlY4B72ZWeEc9GZmhXPQm5kVzkFvZlY4B72ZWeEc9GZmhasV9JKOkbRM0nJJc5qUv1PSryT9XtKnGspWSrpH0p2Suoeq4WZmVs+IwRaQ1AVcBBwF9ACLJS2MiPsqiz0FnA0c389mjoiIJzexrWZmthHq9OgPAJZHxIqIWAfMB2ZWF4iINRGxGHilDW00M7NNUCfoxwGrKtM9eV5dAVwvaYmkWa00zszMNt2gQzeAmsyLFuo4JCJWS9oduEHSAxFx85sqSQ8CswAmTpzYwubNzGwgdXr0PcCEyvR4YHXdCiJidf6/BlhAGgpqttzFETEtIqaNHTu27ubNzGwQdYJ+MTBZ0r6StgNOBBbW2bikkZJG9d0Gjgbu3djGmplZ6wYduomI9ZJmA9cBXcBlEbFU0pm5fK6kPYFuYGfgNUnnAFOAMcACSX11zYuIa9uyJ2Zm1lSdMXoiYhGwqGHe3MrtJ0hDOo2eA/bflAaamdmm8SdjzcwK56A3Myucg97MrHAOejOzwjnozcwK56A3Myucg97MrHAOejOzwjnozcwK56A3Myucg97MrHAOejOzwjnozcwK56A3Myucg97MrHAOejOzwjnozcwK56A3Myucg97MrHAOejOzwjnozcwK56A3Myucg97MrHAOejOzwtUKeknHSFomabmkOU3K3ynpV5J+L+lTraxrZmbtNWjQS+oCLgJmAFOAkyRNaVjsKeBs4Ksbsa6ZmbVRnR79AcDyiFgREeuA+cDM6gIRsSYiFgOvtLqumZm1V52gHwesqkz35Hl11F5X0ixJ3ZK6e3t7a27ezMwGUyfo1WRe1Nx+7XUj4uKImBYR08aOHVtz82ZmNpg6Qd8DTKhMjwdW19z+pqxrZmZDoE7QLwYmS9pX0nbAicDCmtvflHXNzGwIjBhsgYhYL2k2cB3QBVwWEUslnZnL50raE+gGdgZek3QOMCUinmu2bpv2xczMmhg06AEiYhGwqGHe3MrtJ0jDMrXWNTOzzvEnY83MCuegNzMrnIPezKxwDnozs8I56M3MCuegNzMrnIPezKxwDnozs8I56M3MCuegNzMrnIPezKxwDnozs8I56M3MCuegNzMrnIPezKxwDnozs8I56M3MCuegNzMrnIPezKxwDnozs8I56M3MCuegNzMrnIPezKxwDnozs8LVCnpJx0haJmm5pDlNyiXp67n8bklTK2UrJd0j6U5J3UPZeDMzG9yIwRaQ1AVcBBwF9ACLJS2MiPsqi80AJue/A4Fv5v99joiIJ4es1WZmVludHv0BwPKIWBER64D5wMyGZWYC347kVmC0pL2GuK1mZrYR6gT9OGBVZbonz6u7TADXS1oiaVZ/lUiaJalbUndvb2+NZpmZWR11gl5N5kULyxwSEVNJwztnSTqsWSURcXFETIuIaWPHjq3RLDMzq6NO0PcAEyrT44HVdZeJiL7/a4AFpKEgMzPrkDpBvxiYLGlfSdsBJwILG5ZZCJya331zEPBsRDwuaaSkUQCSRgJHA/cOYfvNzGwQg77rJiLWS5oNXAd0AZdFxFJJZ+byucAi4FhgOfAScHpefQ9ggaS+uuZFxLVDvhdmZtavQYMeICIWkcK8Om9u5XYAZzVZbwWw/ya20czMNoE/GWtmVjgHvZlZ4Rz0ZmaFc9CbmRXOQW9mVjgHvZlZ4Rz0ZmaFc9CbmRXOQW9mVjgHvZlZ4Rz0ZmaFc9CbmRXOQW9mVjgHvZlZ4Rz0ZmaFc9CbmRXOQW9mVjgHvZlZ4Rz0ZmaFc9CbmRXOQW9mVjgHvZlZ4Rz0ZmaFqxX0ko6RtEzScklzmpRL0tdz+d2SptZd18zM2mvQoJfUBVwEzACmACdJmtKw2Axgcv6bBXyzhXXNzKyN6vToDwCWR8SKiFgHzAdmNiwzE/h2JLcCoyXtVXNd28pMnz6d6dOnD3czzLYaI2osMw5YVZnuAQ6sscy4mutaASbN+UntZZ9YsbbldVZecFzLbWqm7wHmpptuGpLtmW0J6gS9msyLmsvUWTdtQJpFGvZh4sSJNZo1uKEKhy2l3uGsu6V6h/H4DFfAD+cDzHDVPZT1ttIpAHhiXno5cM+TL6i9Tn/XcKt1t6oT91lFNM3dNxaQ3gecFxEfzNOfAYiI8yvL/BtwU0RcmaeXAdOBSYOt28y0adOiu7t74/bIzLZ6JTy4tUrSkoiY1qysTo9+MTBZ0r7AY8CJwMkNyywEZkuaTxqaeTYiHpfUW2NdM7MhNVzP3DbXIcFBgz4i1kuaDVwHdAGXRcRSSWfm8rnAIuBYYDnwEnD6QOu2ZU/MzKypQYduhoOHbszMWjPQ0I0/GWtmVjgHvZlZ4Rz0ZmaFc9CbmRXOQW9mVjgHvZlZ4Rz0ZmaF2yzfR58/UfvIMFQ9BnhyK6p3OOv2Pm8ddXufO2efiBjbrGCzDPrhIqm7vw8clFjvcNbtfd466vY+bx48dGNmVjgHvZlZ4Rz0G7p4K6t3OOv2Pm8ddXufNwMeozczK5x79GZmhXPQm5kVzkEPSLpM0hpJ93a43gmSfi7pfklLJX2ig3UfI2mZpOWS5nSq3lz3aElXSXog7/v7OlTv3+bjfK+kKyXt0KF6d5B0u6S7cv1f6FC9fyDpzsrfc5LO6UTduf4uSb+W9ONO1Znr/UQ+x0s7ub+57pWS7snHe/P5UY2I2Or/gMOAqcC9Ha53L2Bqvj0KeBCY0oF6u4CHgLcB2wF3daLeSv1XAGfk29sBoztQ5zjgYWDHPP194LQO7a+AnfLtbYHbgIM6dbwr5/wJ0odqOlXn/wDmAT/uYJ3vAe4F3kL6Bb0bgckdrH8lMKaT57bOn3v0QETcDDw1DPU+HhF35NvPA/eTAqndDgCWR8SKiFgHzAdmdqBeJO1MemC9FCAi1kXEM52om3TH31HSCFIQrO5EpZG8kCe3zX+dfhfEkcBDEdGRT5xLGg8cB1zSifoq3gXcGhEvRcR64BfAhzvchs2Og34zIWkS8Iek3l67jQNWVaZ76MwDDKRnEb3At/LT+kskjWx3pRHxGPBV4FHgcdIP2F/f7nr75GGMO4E1wA0R0YnzXHUicGUH6/sa8GngtQ7WCak3f5ik3SS9hfRb1hM6WH8A10taImlWB+sdkIN+MyBpJ+CHwDkR8Vwnqmwyr1M9zBGkYbJvRsQfAi8CbX+NQNJbSc9a9gX2BkZK+mi76+0TEa9GxHuB8cABkt7TqbolbQf8CfCDDtX3IWBNRCzpRH1VEXE/8BXgBuBa0rDk+g424ZCImArMAM6SdFgH6+6Xg36YSdqWFPLfi4irO1RtDxv2csbToWGMXHdPpUd7FSn42+0DwMMR0RsRrwBXAwd3oN4N5GGqm4BjOljtDOCOiPhth+o7BPgTSStJw4Lvl/TdDtVNRFwaEVMj4jDSkOxvOlj36vx/DbCANEw67Bz0w0iSSGPV90fEv3Sw6sXAZEn75t7eicDCTlQcEU8AqyT9QZ51JHBfB6p+FDhI0lvycT+S9JpI20kaK2l0vr0j6UHngU7UnZ1EB4dtIuIzETE+IiaRrq2fRUTHnj1J2j3/nwj8KR3ad0kjJY3quw0cTRpKGnYjhrsBmwNJVwLTgTGSeoDPR8SlHaj6EOAU4J48fgtwbkQsamelEbFe0mzgOtK7MS6LiKXtrLPBfwe+lx9kVgCnt7vCiLhN0lXAHaSn8r+mcx9V3wu4QlIXqXP1/YjoyFsO8zj1UcDHO1HfZuKHknYDXgHOioinO1TvHsCC1I9gBDAvIq7tUN0D8lcgmJkVzkM3ZmaFc9CbmRXOQW9mVjgHvZlZ4Rz0ZmaFc9CbmRXOQW9mVrj/D6uHv27CXpccAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "imp_vals, imp_all = feature_importance_permutation(\n",
    "    predict_method=svm.predict, \n",
    "    X=X_test,\n",
    "    y=y_test,\n",
    "    metric='accuracy',\n",
    "    num_rounds=10,\n",
    "    seed=1)\n",
    "\n",
    "\n",
    "std = np.std(imp_all, axis=1)\n",
    "indices = np.argsort(imp_vals)[::-1]\n",
    "\n",
    "plt.figure()\n",
    "plt.title(\"SVM feature importance via permutation importance\")\n",
    "plt.bar(range(X.shape[1]), imp_vals[indices],\n",
    "        yerr=std[indices])\n",
    "plt.xticks(range(X.shape[1]), indices)\n",
    "plt.xlim([-1, X.shape[1]])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 2 -- Feature Importance for Regressors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.43309137,  0.22058866,  0.00148447,  0.01613953, -0.00529505])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from mlxtend.evaluate import feature_importance_permutation\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.datasets import make_regression\n",
    "from sklearn.svm import SVR\n",
    "\n",
    "\n",
    "X, y = make_regression(n_samples=1000,\n",
    "                       n_features=5,\n",
    "                       n_informative=2,\n",
    "                       n_targets=1,\n",
    "                       random_state=123,\n",
    "                       shuffle=False)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.3, random_state=123)    \n",
    "\n",
    "svm = SVR(kernel='rbf')\n",
    "svm.fit(X_train, y_train)\n",
    "\n",
    "imp_vals, _ = feature_importance_permutation(\n",
    "    predict_method=svm.predict, \n",
    "    X=X_test,\n",
    "    y=y_test,\n",
    "    metric='r2',\n",
    "    num_rounds=1,\n",
    "    seed=1)\n",
    "\n",
    "imp_vals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAMc0lEQVR4nO3dcaid913H8ffHmwalIgN7YSOJa9BgDNLNec0mE9RhIW2H2XBgqm6oGyFidAOHi/8MZP+0/8gQoiHMMEQxDDZHWCNhuMmEbZrb2RXTLnKJlVwzadbpanEsS/f1j3taTu/uvee52bk5vd++X3DhPM/zy8n3ofTNw3PPeZKqQpK0/X3frAeQJE2HQZekJgy6JDVh0CWpCYMuSU0YdElqYlDQkxxKcjnJUpITaxz/hSTfSPLY6OeD0x9VkrSRHZMWJJkDTgL3AsvAxSTnquqJVUv/sareugUzSpIGGHKFfhBYqqorVXUDOAsc3tqxJEmbNfEKHdgFXB3bXgbeuMa6n03yZeAa8P6qurR6QZKjwFGAO++886f379+/+Ykl6RXs0Ucf/VpVza91bEjQs8a+1c8L+BLw2qp6Lsn9wCeBfd/1h6pOA6cBFhYWanFxccBfL0l6QZL/WO/YkFsuy8Cese3drFyFv6iqnq2q50avzwN3JLnrFmaVJN2iIUG/COxLsjfJTuAIcG58QZJXJ8no9cHR+z4z7WElSeubeMulqm4mOQ5cAOaAM1V1Kcmx0fFTwDuA30lyE/gmcKR8jKMk3VaZVXe9hy5Jm5fk0apaWOuY3xSVpCYMuiQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkprYMesBZu3uE4/MeoRNeeqhB2Y9gqSXKa/QJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUhEGXpCYMuiQ1YdAlqQmDLklNGHRJamJQ0JMcSnI5yVKSExus+5kkzyd5x/RGlCQNMTHoSeaAk8B9wAHgwSQH1ln3MHBh2kNKkiYbcoV+EFiqqitVdQM4CxxeY93vAR8Hnp7ifJKkgYYEfRdwdWx7ebTvRUl2AW8HTm30RkmOJllMsnj9+vXNzipJ2sCQoGeNfbVq+8PAB6rq+Y3eqKpOV9VCVS3Mz88PHFGSNMSQf+BiGdgztr0buLZqzQJwNgnAXcD9SW5W1SenMaQkabIhQb8I7EuyF/hP4Ajwa+MLqmrvC6+TfBT4lDGXpNtrYtCr6maS46x8emUOOFNVl5IcGx3f8L65JOn2GPRvilbVeeD8qn1rhryqfvN7H0uStFl+U1SSmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUhEGXpCYMuiQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1MSgoCc5lORykqUkJ9Y4fjjJ40keS7KY5OemP6okaSM7Ji1IMgecBO4FloGLSc5V1RNjy/4eOFdVleQe4GPA/q0YWJK0tiFX6AeBpaq6UlU3gLPA4fEFVfVcVdVo806gkCTdVkOCvgu4Ora9PNr3EknenuQrwCPAb6/1RkmOjm7JLF6/fv1W5pUkrWNI0LPGvu+6Aq+qv62q/cDbgA+t9UZVdbqqFqpqYX5+flODSpI2NiToy8Cese3dwLX1FlfV54AfTXLX9zibJGkThgT9IrAvyd4kO4EjwLnxBUl+LElGr98A7ASemfawkqT1TfyUS1XdTHIcuADMAWeq6lKSY6Pjp4BfAd6V5NvAN4FfHfslqSTpNpgYdICqOg+cX7Xv1Njrh4GHpzuaJGkz/KaoJDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUhEGXpCYMuiQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhM7Zj2Ats7dJx6Z9Qib9tRDD8x6BGnb8gpdkpow6JLUhEGXpCYMuiQ1YdAlqYlBQU9yKMnlJEtJTqxx/NeTPD76+XyS101/VEnSRiYGPckccBK4DzgAPJjkwKpl/w78fFXdA3wIOD3tQSVJGxtyhX4QWKqqK1V1AzgLHB5fUFWfr6r/Hm1+Edg93TElSZMMCfou4OrY9vJo33reDfzdWgeSHE2ymGTx+vXrw6eUJE00JOhZY1+tuTD5RVaC/oG1jlfV6apaqKqF+fn54VNKkiYa8tX/ZWDP2PZu4NrqRUnuAT4C3FdVz0xnPEnSUEOu0C8C+5LsTbITOAKcG1+Q5EeATwDvrKp/m/6YkqRJJl6hV9XNJMeBC8AccKaqLiU5Njp+Cvgg8MPAnyUBuFlVC1s3tiRptUFPW6yq88D5VftOjb1+D/Ce6Y4mSdoMvykqSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUhEGXpCYMuiQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWpiUNCTHEpyOclSkhNrHN+f5AtJvpXk/dMfU5I0yY5JC5LMASeBe4Fl4GKSc1X1xNiyrwO/D7xtK4aUJE025Ar9ILBUVVeq6gZwFjg8vqCqnq6qi8C3t2BGSdIAQ4K+C7g6tr082rdpSY4mWUyyeP369Vt5C0nSOoYEPWvsq1v5y6rqdFUtVNXC/Pz8rbyFJGkdQ4K+DOwZ294NXNuacSRJt2pI0C8C+5LsTbITOAKc29qxJEmbNfFTLlV1M8lx4AIwB5ypqktJjo2On0ryamAR+CHgO0neBxyoqme3bnRJ0riJQQeoqvPA+VX7To29/i9WbsVIkmbEb4pKUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUhEGXpCYMuiQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTeyY9QCS1nb3iUdmPcKmPfXQA7Me4RXNK3RJasKgS1ITBl2SmhgU9CSHklxOspTkxBrHk+RPR8cfT/KG6Y8qSdrIxKAnmQNOAvcBB4AHkxxYtew+YN/o5yjw51OeU5I0wZBPuRwElqrqCkCSs8Bh4ImxNYeBv6yqAr6Y5FVJXlNVX536xNKInwKRXmpI0HcBV8e2l4E3DlizC3hJ0JMcZeUKHuC5JJc3Ne32chfwtWm/aR6e9jveMs/vFrxMzm9Lzg36n9/LxGvXOzAk6FljX93CGqrqNHB6wN+57SVZrKqFWc+xVTy/7avzuUH/89vIkF+KLgN7xrZ3A9duYY0kaQsNCfpFYF+SvUl2AkeAc6vWnAPeNfq0y5uAb3j/XJJur4m3XKrqZpLjwAVgDjhTVZeSHBsdPwWcB+4HloD/A35r60beNrrfWvL8tq/O5wb9z29dWflgiiRpu/ObopLUhEGXpCYM+pRNekzCdpfkTJKnk/zrrGeZtiR7knw2yZNJLiV576xnmqYk35/kn5N8eXR+fzzrmaYtyVySf0nyqVnPMgsGfYoGPiZhu/socGjWQ2yRm8AfVNVPAG8CfrfZf79vAW+pqtcBrwcOjT6V1sl7gSdnPcSsGPTpevExCVV1A3jhMQltVNXngK/Peo6tUFVfraovjV7/Lyth2DXbqaanVjw32rxj9NPmUxFJdgMPAB+Z9SyzYtCna71HIGibSXI38FPAP814lKka3ZJ4DHga+HRVdTq/DwN/CHxnxnPMjEGfrkGPQNDLW5IfBD4OvK+qnp31PNNUVc9X1etZ+Tb3wSQ/OeORpiLJW4Gnq+rRWc8ySwZ9unwEwjaX5A5WYv7XVfWJWc+zVarqf4B/oM/vQ94M/HKSp1i51fmWJH8125FuP4M+XUMek6CXqSQB/gJ4sqr+ZNbzTFuS+SSvGr3+AeCXgK/MdKgpqao/qqrdVXU3K//ffaaqfmPGY912Bn2Kquom8MJjEp4EPlZVl2Y71XQl+RvgC8CPJ1lO8u5ZzzRFbwbeycrV3WOjn/tnPdQUvQb4bJLHWbn4+HRVvSI/3teVX/2XpCa8QpekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKa+H/B1Phef0ncfAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure()\n",
    "plt.bar(range(X.shape[1]), imp_vals)\n",
    "plt.xticks(range(X.shape[1]))\n",
    "plt.xlim([-1, X.shape[1]])\n",
    "plt.ylim([0, 0.5])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 3 -- Feature Importance With One-Hot-Encoded Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upon one-hot encoding, a feature variable with 10 distinct categories will be split into 10 new feature columns (or 9, if you drop a redundant column). If we want to treat each new feature column as an individual feature variable, we can use the `feature_importance_permutation` as usual. \n",
    "\n",
    "This is illustrated in the example below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing the Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we look at a dataset that consists of one categorical feature (`'categorical'`) and 3 numerical features (`'measurement1'`, `'measurement2'`, and `'measurement3'`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>categorical</th>\n",
       "      <th>measurement1</th>\n",
       "      <th>measurement2</th>\n",
       "      <th>label</th>\n",
       "      <th>measurement3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>F</td>\n",
       "      <td>1.428571</td>\n",
       "      <td>2.721313</td>\n",
       "      <td>0</td>\n",
       "      <td>2.089</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>R</td>\n",
       "      <td>0.685939</td>\n",
       "      <td>0.982976</td>\n",
       "      <td>0</td>\n",
       "      <td>0.637</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>P</td>\n",
       "      <td>1.055817</td>\n",
       "      <td>0.624210</td>\n",
       "      <td>0</td>\n",
       "      <td>0.226</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>S</td>\n",
       "      <td>0.995956</td>\n",
       "      <td>0.321101</td>\n",
       "      <td>0</td>\n",
       "      <td>0.138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>R</td>\n",
       "      <td>1.376773</td>\n",
       "      <td>1.578309</td>\n",
       "      <td>0</td>\n",
       "      <td>0.478</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  categorical  measurement1  measurement2  label  measurement3\n",
       "0           F      1.428571      2.721313      0         2.089\n",
       "1           R      0.685939      0.982976      0         0.637\n",
       "2           P      1.055817      0.624210      0         0.226\n",
       "3           S      0.995956      0.321101      0         0.138\n",
       "4           R      1.376773      1.578309      0         0.478"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "\n",
    "df_data = pd.read_csv('https://gist.githubusercontent.com/rasbt/b99bf69079bc0d601eeae8a49248d358/raw/a114be9801647ec5460089f3a9576713dabf5f1f/onehot-numeric-mixed-data.csv')\n",
    "df_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "df_X = df_data[['measurement1', 'measurement2', 'measurement3', 'categorical']]\n",
    "df_y = df_data['label']\n",
    "\n",
    "\n",
    "df_X_train, df_X_test, df_y_train, df_y_test = train_test_split(\n",
    "     df_X, df_y, test_size=0.33, random_state=42, stratify=df_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we do the one-hot encoding on the categorical feature and merge it back with the numerical columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "ohe = OneHotEncoder(drop='first')\n",
    "ohe.fit(df_X_train[['categorical']])\n",
    "\n",
    "df_X_train_ohe = df_X_train.drop(columns=['categorical'])\n",
    "df_X_test_ohe = df_X_test.drop(columns=['categorical'])\n",
    "\n",
    "ohe_train = np.asarray(ohe.transform(df_X_train[['categorical']]).todense())\n",
    "ohe_test = np.asarray(ohe.transform(df_X_test[['categorical']]).todense())\n",
    "\n",
    "X_train_ohe = np.hstack((df_X_train_ohe.values, ohe_train))\n",
    "X_test_ohe = np.hstack((df_X_test_ohe.values, ohe_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.65747208 0.95105388 0.36       0.         0.         0.\n",
      "  0.         0.         0.         0.         0.         0.\n",
      "  0.         0.         0.         1.         0.         0.\n",
      "  0.         0.         0.        ]\n",
      " [1.17503636 1.01094494 0.653      0.         0.         0.\n",
      "  0.         0.         0.         0.         0.         0.\n",
      "  0.         0.         0.         0.         1.         0.\n",
      "  0.         0.         0.        ]\n",
      " [1.25516647 0.67575824 0.176      0.         0.         0.\n",
      "  0.         0.         0.         0.         0.         0.\n",
      "  0.         0.         1.         0.         0.         0.\n",
      "  0.         0.         0.        ]]\n"
     ]
    }
   ],
   "source": [
    "# look at first 3 rows\n",
    "print(X_train_ohe[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fitting a Baseline Model for Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "\n",
    "pipe = make_pipeline(StandardScaler(),\n",
    "                     MLPClassifier(max_iter=10000, random_state=123))\n",
    "\n",
    "params = {\n",
    "    'mlpclassifier__hidden_layer_sizes': [(30, 20, 10), \n",
    "                                          (20, 10), \n",
    "                                          (20,),\n",
    "                                          (10,)],\n",
    "    'mlpclassifier__activation': ['tanh', 'relu'],\n",
    "    'mlpclassifier__solver': ['sgd'],\n",
    "    'mlpclassifier__alpha': [0.0001],\n",
    "    'mlpclassifier__learning_rate': ['adaptive'],\n",
    "}\n",
    "\n",
    "gs = GridSearchCV(estimator=pipe, \n",
    "                  param_grid=params, \n",
    "                  scoring='accuracy', \n",
    "                  refit=True,\n",
    "                  n_jobs=-1,\n",
    "                  cv=10)\n",
    "\n",
    "gs = gs.fit(X_train_ohe, df_y_train.values)\n",
    "model = gs.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regular Permutation Importance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we compute the feature importance the regular way were each one-hot encoded feature is treated as an individiual variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mlxtend.evaluate import feature_importance_permutation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "imp_vals, imp_all = feature_importance_permutation(\n",
    "    predict_method=model.predict, \n",
    "    X=X_test_ohe,\n",
    "    y=df_y_test.values,\n",
    "    metric='accuracy',\n",
    "    num_rounds=50,\n",
    "    seed=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feat_names_with_ohe = ['measurement1', 'measurement2', 'measurement3'] \\\n",
    "    + [f'categorical_ohe_{i}' for i in range(2, 20)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAFPCAYAAABAjMnjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAmyElEQVR4nO3de7gkVXnv8e+PQQRFhtuoyEVIHBMnHhQcEYOJE4U4iIghGsGjEpJzCAm3aExEzXM4mBtqTI4XAiERhIghRGMy6iRIiICXgAy34S4TEJkwwAAKeDki+OaPqm169vTeVXuv6l5de/0+z1PP7q6u1e9aq6v221W1ukoRgZmZlWuL3BUwM7O8nAjMzArnRGBmVjgnAjOzwjkRmJkVzonAzKxwW+auwHzsvPPOseeee+auhplZr1x99dUPRMSS6fN7mQj23HNP1qxZk7saZma9IumuYfN9aMjMrHBOBGZmhXMiMDMrnBOBmVnhnAjMzArnRGBmVjgnAjOzwhWVCFasWMGKFStyV8PMbKIUlQjMzGxzTgRmZoVzIjAzK5wTgZlZ4ZwIzMwK50RgZlY4JwIzs8I5EZiZFc6JwMyscE4EZmaFcyIwMyucE4GZWeGcCMzMCudEYGZWOCcCM7PCdZIIJK2UdJukdZJOHvK6JH24fn2tpH3r+VtL+pqk6yXdJOnULupjZmbtJScCSYuA04GDgWXAkZKWTVvsYGBpPR0DnFHP/wHwioh4AfBCYKWk/VPrZGZm7XWxR7AfsC4i7oiIx4ALgMOmLXMYcF5UrgC2l7RL/fw79TJPqqfooE5mZtZSF4lgV+Dugefr63mtlpG0SNJ1wP3AxRFx5bAgko6RtEbSmo0bN3ZQbTMzg24SgYbMm/6tfsZlIuKJiHghsBuwn6TnDwsSEWdFxPKIWL5kyZKU+pqZ2YAuEsF6YPeB57sB98x1mYj4NnApsLKDOpmZWUtdJIKrgKWS9pK0FXAEsGraMquAt9ajh/YHHo6IDZKWSNoeQNI2wIHArR3UyczMWtoy9Q0i4nFJxwMXAYuAsyPiJknH1q+fCawGXg2sA74HHF0X3wU4tx55tAVwYUR8LrVOZmbWXnIiAIiI1VT/7AfnnTnwOIDjhpRbC+zTRR3MzGx+/MtiM7PCORGYmRXOicDMrHBOBGZmhXMiMDMrnBOBmVnhnAjMzArnRGBmVjgnAjOzwjkRmJkVzonAzKxwTgRmZoVzIjAzK5wTgZlZ4ZwIzMwK50RgZlY4JwIzs8I5EZiZFc6JwMyscE4EZmaFcyIwMyucE4GZWeGcCMzMCudEYGZWOCcCM7PCORGYmRXOicDMrHCdJAJJKyXdJmmdpJOHvC5JH65fXytp33r+7pK+KOkWSTdJOqmL+piZWXvJiUDSIuB04GBgGXCkpGXTFjsYWFpPxwBn1PMfB34nIp4H7A8cN6SsmZmNUBd7BPsB6yLijoh4DLgAOGzaMocB50XlCmB7SbtExIaIuAYgIh4FbgF27aBOZmbWUheJYFfg7oHn69n8n3njMpL2BPYBrhwWRNIxktZIWrNx48bUOpuZWa2LRKAh82Iuy0jaFvg08NsR8ciwIBFxVkQsj4jlS5YsmXdlzcxsU10kgvXA7gPPdwPuabuMpCdRJYHzI+IfOqjPRFqxYgUrVqzIXQ0zs810kQiuApZK2kvSVsARwKppy6wC3lqPHtofeDgiNkgS8DHgloj4sw7qYmZmc7Rl6htExOOSjgcuAhYBZ0fETZKOrV8/E1gNvBpYB3wPOLoufgDwFuAGSdfV894dEatT62VmZu0kJwKA+h/36mnzzhx4HMBxQ8p9meHnD8zMbEz8y2Izs8I5EZiZFc6JwMyscJ2cI5hEe578+c3m3XvHgzO+9o3TDhl5nczMJpH3CMzMCudEYGZWOCcCM7PCORGYmRXOicDMrHBOBGZmhXMiMDMrnBOBmVnhnAjMzArnRGBmVjgnAjPLwnftmxxOBGZmhXMiMDMrnBOBmVnhnAjMzArnRGBmVjgngpY8wsHMFionAjOzwjkRmJkVzonAzKxwTgRmZoVzIjAzK5wTgZlZ4TpJBJJWSrpN0jpJJw95XZI+XL++VtK+A6+dLel+STd2URczM5ub5EQgaRFwOnAwsAw4UtKyaYsdDCytp2OAMwZe+ziwMrUeC1kff8PQxzqblaqLPYL9gHURcUdEPAZcABw2bZnDgPOicgWwvaRdACLicuChDuphZmbz0EUi2BW4e+D5+nreXJcx+zHvUZiNTxeJQEPmxTyWmT2IdIykNZLWbNy4cS5Fi+Z/qGbWpItEsB7YfeD5bsA981hmVhFxVkQsj4jlS5YsmVdFzcxsc10kgquApZL2krQVcASwatoyq4C31qOH9gcejogNHcQ2M7NEyYkgIh4HjgcuAm4BLoyImyQdK+nYerHVwB3AOuCvgN+aKi/pb4F/B35K0npJv55aJzMza2/LLt4kIlZT/bMfnHfmwOMAjpuh7JFd1MHMzOank0Sw0Ox58uc3m3fvHQ/O+No3Tjtk5HUyMxsVJ4IRcCIxsz7xtYbMzArnRGBmVjgnAltw/CM6s7lxIjAzK5wTgc3I36znxv1lfeVEYGZWOCcCM7PCORGYmRXOicDMrHBOBGZmhXMiMBvgkT9WIicCM7PC+aJzE8YXrDOzcfMegZnNmw+lLQzeI2jpmW86LXcVzMxGwnsEZmaFcyIwmxA+zGK5OBGYmRXO5wgWEI84svmY2gu59NJLs9bD8vEegZn1kg+ldceJwMyscD40ZEDaYaW5lp1e3szy8h6BmRXHh5U25T0Cy84nuc3y8h6BmVnhnAjMzArXSSKQtFLSbZLWSTp5yOuS9OH69bWS9m1b1szMRiv5HIGkRcDpwEHAeuAqSasi4uaBxQ4GltbTS4AzgJe0LGs2o3GOdvK5CVuoujhZvB+wLiLuAJB0AXAYMPjP/DDgvIgI4ApJ20vaBdizRdnO+Aqi1hUnEVtIVP1vTngD6fXAyoj4X/XztwAviYjjB5b5HHBaRHy5fn4J8E6qRDBr2YH3OAY4BmCPPfZ40V133ZVU73FL+Rl/aWVzxu5rm9samsA+WR2RHfZFqXEPakRlc8ZuKjubXGWnl5+JpKsjYvn0+V2cI9CQedOzy0zLtClbzYw4KyKWR8TyJUuWzLGKZmY2ky4ODa0Hdh94vhtwT8tltmpR1szMRqiLRHAVsFTSXsB/AkcAb5q2zCrg+PocwEuAhyNig6SNLcqaWUeGHT5YccUHALjU5zGKlZwIIuJxSccDFwGLgLMj4iZJx9avnwmsBl4NrAO+Bxw9W9nUOplZ95xE8ms6rzJfnVxiIiJWU/2zH5x35sDjAI5rW9Y25evEm9ko+ZfFZmaFcyIwMyucrz5qZiO3kM4vjOo4fU5OBDYjn5swK4MTwZj4n6qZTSqfIzAzG5N7P3nyjw8tTRInAjOzwvnQkJlNvIV0snkSOREscD430R/+rCwXHxoyMyuc9wjMBpT4rXyht9mHlZp5j8DMrHBOBGZmhXMiMDMrnBOBmVnhnAjMzArnUUO24Cz0UTBmXXMisJEo8Z+x22x95UNDZmaF8x6BTSR/0zQbH+8RmJkVzonAzKxwPjRkZjaDUq5T5D0CM7PCORGYmRXOh4bMzEagT4eVnAjMzCbMsCQCo0skSYeGJO0o6WJJt9d/d5hhuZWSbpO0TtLJA/PfIOkmST+StDylLmZmNj+p5whOBi6JiKXAJfXzTUhaBJwOHAwsA46UtKx++UbgcODyxHqYmdk8pR4aOgxYUT8+F7gUeOe0ZfYD1kXEHQCSLqjL3RwRt9TzEqthZqXxr8+7k7pH8IyI2ABQ/336kGV2Be4eeL6+nmdmZhOgcY9A0r8Czxzy0ntaxhj2dT9alh2sxzHAMQB77LHHXIubmdkMGhNBRBw402uS7pO0S0RskLQLcP+QxdYDuw883w24Z64VjYizgLMAli9fPudEYmY2xYeVNpV6aGgVcFT9+Cjgn4YscxWwVNJekrYCjqjLmZnZBEhNBKcBB0m6HTiofo6kZ0laDRARjwPHAxcBtwAXRsRN9XK/JGk98FLg85IuSqyPmZnNUdKooYh4EHjlkPn3AK8eeL4aWD1kuc8An0mpg5mZpfG1hszMCudEYGZWOCcCM7PCORGYmRXOicDMrHBOBGZmhfP9CMzM5mAh/irZewRmZoVzIjAzK5wPDZmZjcmkHlbyHoGZWeGcCMzMCudEYGZWOCcCM7PCORGYmRXOicDMrHBOBGZmhXMiMDMrnH9QZmbWE6P6QZr3CMzMCudEYGZWOCcCM7PCORGYmRXOicDMrHBOBGZmhXMiMDMrnBOBmVnhnAjMzAqXlAgk7SjpYkm31393mGG5lZJuk7RO0skD8z8g6VZJayV9RtL2KfUxs/649NJLJ/bWjaVJ3SM4GbgkIpYCl9TPNyFpEXA6cDCwDDhS0rL65YuB50fE3sDXgXcl1sfMzOYoNREcBpxbPz4XeN2QZfYD1kXEHRHxGHBBXY6I+EJEPF4vdwWwW2J9zMxsjlITwTMiYgNA/ffpQ5bZFbh74Pn6et50vwb880yBJB0jaY2kNRs3bkyospmZDWq8+qikfwWeOeSl97SMoSHzYlqM9wCPA+fP9CYRcRZwFsDy5ctjpuXMzGxuGhNBRBw402uS7pO0S0RskLQLcP+QxdYDuw883w24Z+A9jgJeA7wyIvwP3sxszFIPDa0CjqofHwX805BlrgKWStpL0lbAEXU5JK0E3gm8NiK+l1gXMzObh9REcBpwkKTbgYPq50h6lqTVAPXJ4OOBi4BbgAsj4qa6/EeBpwEXS7pO0pmJ9TEzszlKukNZRDwIvHLI/HuAVw88Xw2sHrLcc1Lim5lZOv+y2MyscOrj+VlJG4G75ll8Z+CBDGVzxu5j2Zyx3eZ+lM0Zu69tfnZELNlsbkQUNQFrcpTNGbuPZftab7fZ/TXJbZ5p8qEhM7PCORGYmRWuxERwVqayOWP3sWzO2G5zP8rmjN3XNg/Vy5PFZmbWnRL3CMzMbIATgZlZ4ZwIzMwK50RgIyNp2P0pFjRJO+Wug9lcORHMgaTzxhhrP0kvrh8vk/R2Sa9uUW5bSe+VdJOkhyVtlHSFpF9tGfcnJb1D0ockfVDSsZIWtyi347RpJ+BrknaQtGOb2LO894w3LBpY5qclvVLSttPmr2wot52kP5H0N5LeNO21v2goe5qknevHyyXdAVwp6S5JL29R52sk/b6kn2xadkjZxXX8WyU9WE+31PO2byj7Eknb1Y+3kXSqpM9Kel/TZy3pREm7z7bMqEjaStJbJR1YP3+TpI9KOk7SkxrKPlPSGZJOl7STpP8r6QZJF9aX0J9LPV5Wb4+/2GLZlQOPF0v6WH2P9k9KekZD2eWSvijpE5J2V3Vf+IclXSVpn7nUucmCTgT1P4d/lvT5+h/cxyV9W9LXJD2voeyqadNngcOnnjeU3V3SBZK+JOndgyuppH9sUe9TgA8DZ0j6E6qrtG4LnKzqJj6zOR+4A3gVcGr9Pm8BfkHSHzfEPRE4E9gaeDGwDdW9JP5d0oqGuA8AVw9Ma6juRHdN/XhWkvadYXoR8MIW9f4n4ATgRkmHDbw8a5uBc6hunvRp4AhJn5b05Pq1/RvKHhIRUz/1/wDwxqgupHgQ8MGGsgA7ANsDX6zXybdJelaLcgAXAt8CVkTEThGxE/AL9by/byh7NjB12fcPAYuB99Xzzmko+wdUye5Lkn5L0uaXK5gHSTe0WOwc4BDgJEl/A7wBuJJqXf3rhrIfB26mulviF4Hv1+/1Jap1fra6fW3g8f/mv6+afIqkze7TPs3g+vdBYANwKNXl+f+yoexfAO8HPg98FfjLiFhMdW/4Wb+kzFnXP1WepAm4vO70I6muTXQE1UZ/KHBJQ9lrgE8AK4CX13831I9f3lD2YuBYqn9gH6k/xJ3q165tUe8bgEXAU4BHgO3q+dsAaxvKXj/t+VX13y2AW9vErR8/Bbi0frxHU72BdwD/AvyPgXl3zuGzegL4N6qNdPr0/Rb13rZ+vCdV4jmpTX8D1017/h7gK8BOwDUNZW8FtqwfXzG9Ti3afM3A45+j2rjvrdt8TEPZ2+bzWv36LcPqMKw/hpS9tl6XfhH4GLCx/tyPAp7WUPbwGaZfBja26K+19d8tgfsG1lW12C6uHXj8zbm2eeDxVcCS+vFTmz7naZ/x9HVtLnGn1/na2crOdUq6DHUPPC0iPgsg6Q8i4oJ6/mclndpQdjlwEtU/ht+NiOskfT8iLmsRd0lETH3LOEHSm4HLJb2WabfpnMHjEfEE8D1J/xERjwBExPcl/aih7HclvSwivizpUOChuuyPJA27beh0W1L9U34y1bceIuKbTbveEfGnki4A/lzS3cAptGvrlFuA34iI26e/UL/fbBZFxHfqenyj3nv5lKRnM/xWqYOeLGmLiPhRXf6PJK2n+hKx7exFOR1YLek04F8k/T/gH6guzX5dQ9lNRMSXgC9JOoFqj+KNzP7Dobsk/R5wbkTcB1AfavhVNr1H+DA3Sjo6Is4Brpe0PCLWSHou8MPmqsaPgC8AX6jXi4Opvmz9KTDbHsLfUe2xDlsvtm6IC7CFqptbPZXqi8piqvX7ycCs6yebHv2Yfoi36cjIFpJ2qJdTRGwEiIjvSnq8oezTJb2daj3cTpKi/k/eIu7/rw8/LQZC0usi4h9VHXZ8oqHs3HSZVSZtYuBbAvBb0167seV77Ea1q/1RpmXlWcrcBGw9bd6BwDpgQ4vyVwJPqR9vMTB/Mc3fUvcGvgZ8G/gy8Nx6/hLgxIayJwFrqf4B3QocPVD28jn0+6HAFcC9cyjzeuCnZnjtdQ1l/w144bR5W1Jt8E80lH0/cOCQ+SuB21vUewXVP7hrqfZMVgPHAE9qUfaC+azXddkdqA7n3Er1z/AhqmT6PmDHhrKLqQ6V/Ee9rv2Q6nDiZcALGspeO8tr2zSUvRp4/gyv3d2izW+r63kXcCJwCfBXdb+f0lD2vdR7jdPmPwf4VEPZb9Rx76z/PrOevy3N3+pPmTZN7U08EzivoewLqG7o9c/AT1Mdxvs21f+XA+a77gybFvQviyX9BnB+1N8WB+Y/Bzg+In57Du91CFXnv3va/B0i4lvT5r2N6h/2ZdPm7wO8PyIOaoj15Ij4wZD5OwO7RMQNM8VOJelngOdRJcpbZ1imMa6kbYCfjIgbu6zfDLF2o9qLunfIawdExFfqx533V59JehrwE1RJc33UexYDrw9bt58bEV9v8d7Dyv4ccFdEfHPI8ssjos25pGdBdfMrVSfFD6T6gjZ4HH8sn7OkpwDPiIg7xxl3JLrMKpM6MSR7Dps3z/ee8Rv6KOM2xZ5h+aP7GLeP9c7Z5lz91bast4vJ+YynpgU9amjAR1rOm4/ZjkGPMm5T7GGazotMatw+1jtnm3P1V9uy3i7S43ZVZyDxnsWTTtJLgZ8FltQnbKZsRzUqpwubHVsbU9yZYq+dYVkBs45bzhm3j/XO2eZc/ZVS1tvF3OKOqc7AAk8EwFZUJ3S2pB4BU3uE6uTkQosL1QryKqrx5INENYx1UuP2sd4525yrv1J4u5hb3LHVeUEngqhO1l4m6eMRMd97HDfZbHdwTHGHxgY+RzU64rrNFpYuneC4fax3zjbn6q95l/V2Mee446hz9X71iYcFrR4f/Q6qHxv9OPlFxCtaln8ZsDQizql/Sblt/PdIgR0j4qFRxE2J3eJ9Zx3hkCtuavlJrPeo25xSNqW/vF0sjPULKGbU0PXAbwL7AS+amlqWPQX4LPD1+vmzgK+MOm5q7BbvPduojixxF2q9R9nmXP3l7WLhrF8RC/+XxVMej4gz5ln2l4B9qC45QVTjl582e5FO4qbGbjLbbn+uuKnlJ7Xeo2xzStmU/vJ2sXDWr2KGj35W1QWydtHAFTJbln0sqrQbAJKeOqa4qbGbzHZMMFfc1PKTWu9RtjmlbEp/ebtYOOtXMXsER9V/f3dgXlD9qrLJhZL+Ethe1ZUHf43qZ+2jjpsaO0WuuKn6Wu9cUvrL28UCWr+KOFmcStJBVFdbFHBRRFzc99iSro2IGa9pnituavlJrPeo25xSNqW/vF0sjPULKOZk8VOA3wfOqp8vBV6zUOMOxH8Zm144bq+B12a9MFnOuH2sd8425+qvhH72djFhn3H2lWJMH/zfAb9HfcVRquv6X9ey7OHA7cDDVD98eRR4ZNRxO4idMqojS9y+1jtzm3P1l7eLnqxfrWJ0+WaTOgFr6r/XDsy7vmXZdcDzxh23g9jXUe2+Dsae9eYdueP2td6Z25yrv7xd9GT9ajOVMmroMVWXRQ4AVfeI3ewyzzO4LyJuyRA3OXZUa8xU7LmMcMgVN7V8rnrnbHOu/vJ2Mb64oxytBJQzaugUqtvp7S7pfOAAqjs5zUjS4fXDNZL+DvhHBlbWiPiHUcTtMPacRzjkirsA6j32NqeUTekvbxe9XL8aFTNqSNJOVDcjF9X9ZR9oWP6cWV6OiPi1UcTtOPacRjjkiptafhLqPe42p5RN6S9vF+OL21XZVu9fUCLYm82vbdImk/cyrtkk83YxWYo4RyDpbOBs4Jep7qd7KPCalmV3k/QZSfdLuk/Sp1XdGnGkcTuIfbik2yU9LOkRSY9KemSS4/a13pnbnKu/vF30ZP1qpcszz5M6ATcnlL0YOJrqm8uWVMcyLx513A5ip4xwyBK3r/XO3OZc/eXtoifrV6sYo3zzSZmAjwHL5ln2ujbzuo7bQex5jzPOFbev9c7c5lz9lVLW28WYPuO2Uymjhs4F/l3SvVRn+0V1kmfvFmUfkPRm4G/r50cCD44h7rxiq5sRDmOP28d652xzrv7qqKy3ixZxO6pzK0WcLJa0Dng7cAPwo6n50eIuSZL2AD4KvJRqHO9XgZNalp133PnGVgcjHHLE7WO9c7Y5V391VNbbRYu4XdS5tVHvckzCBPzbCN/7XTniNsVeiHFLrXcf+8vbRb+mIkYNAbdK+qSkI+sz8IcP7HalekOmuLPGThnhkDNuH+uds825+iuxrLeLyfmMgUKGj1Jd1OoHVD/ImPNwtQaz3R1olHGbYp8DrKK6QNWuVBetmm1Xc1Li9rHeOducq79Synq7mJzPuJJ7l6TvEx3cL3QUsUkY4ZAzbh/rnbPNufprlGUn9XPOFXeUdZ6aitgjkPRcSZdIurF+vrek3+/q7TPFnTU29QgHSYvq6c20H9WRM24f652zzbn6a95lvV1M1GdcyZW1x/wN4TJgPza9jOuNHb33u3PEbRF7D6rdyY3A/VRDz5496XH7WO+cbc7VX4l97e1iQj7jqamU4aNXRcSLNXBLN0nXRcQLZynzEWa5KXREnDiKuF3FblG3d0XEn0xC3NTyk17vUbQ5pWxKf3m7GF/ccZSdUsoPyh5Qdc3zAJD0emBDQ5k1meJ2FbvJG4DpK0+uuKnlJ73eo2hzStmU/vJ2Mb644ygLlJMIjgPOAn5a0n8CdwL/c7YCEXFujrgdxm6y2fHMXHFTy/eg3p23OaVsSn95uxhf3DGVBQpIBJIWAb8ZEQequrPPFhHx6BzKLwHeCSwDtp6aHxGvGGXclNgtzbirmytuavkJrvfI2pxSNqW/vF0sqPVr4Y8aiogngBfVj78715UOOB+4BdgLOBX4BnDVGOLOO3ZLs32LyBU3tfyk1nui9ggGpPSXt4uFs34t/ERQu1bSKklvmccvGXeKiI8BP4yIy6K6vsf+Y4ibGrvJ309g3NTyk1rvUbY5pWxKf3m7WDjr18I/NFTbkWrc7eAuXABtrt73w/rvBkmHAPcAbX/enRJ3XrHbjnCIiD+epLh9rHfONufqr47KertoEbejOrdSRCKIiKMTiv+hpMXA7wAfAbYD3jaGuPON3cUIhxxx+1jvnG3O1V/JZb1dtDaO0UpAOZehPochmTW6vIzrBMU1m2TeLiZPEXsEwOcGHm8N/BLVbl0jSedSXXP82/XzHYAPtlxp5x03NXbiiJAscfta78xtztVf3i56sn61UcTJ4oj49MB0PvArwPNbFt976oOv3+tbwD5jiJsUm7QRDrnippbPVe+cbc7VX94u+rN+NYsOr1fRlwn4KWBdy2WvB3YYeL4jcMOo46bGBq6u/64dmHfZJMfta70ztzlXf3m76Mn61WYq4tCQpEfZ9JjkvVS7Wm18EPiqpE/Vz98A/NEY4ibFJm1UR664qeVz1Ttnm3P1l7eL/qxfzbrMKgt1ojo2dzxwArCsD7GpbvSxmGqX+4vA1cBrexC3d/XO2eZc/ZVaNnUqabtIrXObqZRRQwdQ3cjhu6qu5b0v8KGY/abT20XEI5J2HPZ6RDw0irhdxZ6PXHFT9bXeuaT0l7eL8cUdq3Fm8VwTsJbqZ9gvqB+fRMMxNuBz9d87gTsGpjuBO0YVt8PY5wLbDzzfATh7EuP2vd452pyrv7xd9G/9ahWjyzeb1In6FnLA/wF+fXBeQzkBe4w7bkexr20zb1Li9rneudqcq7+8XfRr/WozFTF8FHhU0ruANwOfr6+A+KSmQlH1+GfGHbej2FvU45wBqHdrGwcH5IqbWj5zvbO0OaVsSn95uxhf3A7KtlLEqCHgjcCbqL593CtpD+ADLcteIenFETGfcbspcVNjp4xwyBU3tXyueudsc67+8nYxvrip61ejIk4Wp5B0M/Bc4C7gu1S7iBERe096bEnLqC7sJeCSiLh5kuP2ud652pxSNqW/vF30a/1qfP8SEoGk/akuEvU8YCtgEfCdiFjcouyzh82PhhEOqXHnG7ujUR1jj9vHeudsc67+6qist4sWccc5WqmURLAGOILqut3LgbcCSyPi3S3LvwD4ufrplyLi+nHEnU9sSZ+LiNdIupNNf7Qz9e3lJyYxbh/rnbPNufqri7LeLtrF7arOrUSHZ54ndQLW1H8Hf6L91ZZlTwJuBN5bTzcAJ4w6bkps0kc45Irbu3rnbHOu/uqgrLeLMX3GreOMOsAkTMDlVLug5wHvp7qG+PUty64Fnjrw/KmDK/Co4nYQ++qE/soSt6/1ztzmXP3l7aIn61ebqZTho2+hutLq8VQneXYHfrllWQFPDDx/op436ripsa+Q9OI5xJqEuKnlc9U7Z5tz9Ze3i/6sX42KGD4aEXdJ2gbYJSJOnWPxc4ArJU2NIX4d8LExxE2KDfwC8BuS5jPCIVfcvtY7Z5tz9Ze3i/6sX41KOVl8KPCnwFYRsZekFwLvjYjXtiy/L/Ayqg/g8oi4dhxxE2PPe1RH5ri9q3fONufqr5Sy3i7mFje1zm2UkgiuphqDe2lE7FPPW9smo84wdOvRiPjhkPmdxU2NXZef76iOLHFTy+esd642p5RNXLe9XYwpbmrZNko5R/B4RDw8z7LXABuBrwO314/vlHSNpBeNMG5SbEknUd3Z6On19AlJJ0x43F7WO2ebc/VXYllvF2Ncv1oZ9dnoSZiojuO9ieqs/1KqH7Oc2bLsmcCrBp7/IvBnwP7AlaOK20HslBEOWeL2td6Z25yrv7xd9GT9ajOVskdwAvAzwA+ATwIPU40JbmN5RFw09SQivgD8fERcATx5hHFTY6eMcMgVN7V8rnrnbHOu/vJ20Z/1q1ERo4ao7iq0jKq9WwKHAa8F2hyTfEjSO4EL6udvBL6l6oqJPxph3NTYKSMccsVNLZ+r3jnbnKu/vF30Z/1qVMrJ4tuAd1D9KvDHH1i0O2O/M3AK1UgBgC9T/arwYapf/K0bRdzU2HX5+Y5wyBK3z/XO1eaUsonrtreLHq1fje9fSCL4ckS8rHnJWd9j24j4zrjjJsROGuGQK24f652zzbn6K6Wst4u5xe2izo26POEwqRPwSuCvgSOBw6emlmV/FrgZ+Gb9/AXAX4w6bgexv0F1LPEB4MH68XqqkQ8vmsS4fa135jbn6i9vFz1Zv1rVr4s3mfQJ+ASwhuren+fUU9v7hV5J9RP4awfm3TjquB3EThnhkCVuX+uduc25+svbRU/Wr1b16+JNJn0Cbkgoe2X9d/DDb3vBunnH7SD2mpnmAddNYty+1jtzm3P1l7eLnqxfbaZSRg1dIWlZzO+uPndL+lkgJG0FnAjcMoa4qbFTRjjkitvXeudsc67+8nbRn/WrWRfZZNInqg/rMeA2qh9n3ED7H3PsTPWrvvuA+6l2a3ccddwOYu9M9UOda+vpI8ASqsv/PmcS4/a13hPQ5lz95e2iB+tXm6mUUUPPHjY/2g0fPSAivtI0r+u4qbEHlp/PCIcscVPL5653jjanlE1ct71djCluF2UbdZFNFvIEXNNm3qTFJm2EQ5a4fa135jbn6i9vFz1Zv9pMpZwjmDNJL6X6AJZIevvAS9tR3Wx70mP/OfAqYBVARFwv6ecnMe4CqPfY25xSNqW/vF30cv1q5EQws62Aban66GkD8x8BXt+H2BFxt7TJJUmemGnZzHFTy2evd4Y2p5RN6S9vF+OL20nZNpwIZhARlwGXSfp4dHgDiDHGnvMIh1xxF0C9x97mlLIp/eXtopfrV6MiThankLQE+D2qqyVuPTU/Il4xybFVXRflQ8CBVNcn+QJwYkQ8NKlx+1rvzG3O1V/eLsYXN2n9aqXLEw4Lcao7/depMvDLgbOB9016bOCANvMmKW5f6525zbn6y9tFT9avVjHG8cH1eQKurv+uHZh32aTHJm2EQ5a4fa135jbn6i9vF+OLO/IRWj5H0GzqCn8bJB0C3APsNqmxOxrhMPa4fax3zjbn6q+OyqYqYrsY5wgtJ4JmfyhpMfA7VL/o2w747QmO3cUIhxxx+1jvnG3O1V9dlE1VynYxvhFa49iV6/NEdYXE7Qee78gcrpSYKzbw7L7F7Wu9M7c5V395u+jJ+tVm8h5Bs70j4ttTTyLiIUn79CD29yR9gPmN6sgVN7V8rnrnbHOu/vJ20Z/1q1EpN69PsYWkHaaeqLpb0LgSaErs84Fbgb2AU6lubnHVhMdNLZ+r3jnbnKu/vF30Z/1qNo5duT5PwFuphov9AdX9SW8F3jLpsUkb4ZAlbl/rnbnNufrL20VP1q82kw8NNYiI8yStAV5B9WOOw2P+11EfZ+x5j+rIFTe1fMZ6Z2tzStmU/vJ20av1q9k4Mrin8U/Aa4DFwPOBLwJXA4dOetw+1jtnm3P1V1+nEtevVjFyfzCeRjORaVRHatw+1jtnm3P1V1+nEtevNpNPFi9cm41wAMYxqiM1bh/rnbPNufqrr0pcvxo5ESxcuUZ1pMbtY71ztjnn6J0+KnH9auQVZuH6IPBVSZ8CAvgV4I96ELeP9c7Z5lz91Vclrl+NfBnqBUzSMv57hMMlMaZRHalx+1jvnG3O1V99VeL61fj+TgRmZmXzOQIzs8I5EZiZFc6JwMyscE4EZmaFcyIwMyvcfwFMEeXhWAx56QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "std = np.std(imp_all, axis=1)\n",
    "indices = np.argsort(imp_vals)[::-1]\n",
    "\n",
    "plt.figure()\n",
    "#plt.title(\"Feature importance via permutation importance w. std. dev.\")\n",
    "plt.bar(range(len(feat_names_with_ohe)), imp_vals[indices],\n",
    "        yerr=std[indices])\n",
    "plt.xticks(range(len(feat_names_with_ohe)),\n",
    "           np.array(feat_names_with_ohe)[indices], rotation=90)\n",
    "plt.xlim([-1, len(feat_names_with_ohe)])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, notice that if we have a lot of categorical values, the feature importance of the individual binary features (after one hot encoding) are now hard to interpret. In certain cases, it would be desireable to treat the one-hot encoded binary features as a single variable in feature permutation importance evaluation. We can achieve this using feature groups."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### With Feature Groups"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the example below, all the one-hot encoded variables are treated as a feature group. This means, they are are all shuffled and analyzed as a single feature inside the feature permutation importance analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_groups = [0, 1, 2, range(3, 21)]\n",
    "\n",
    "imp_vals, imp_all = feature_importance_permutation(\n",
    "    predict_method=model.predict, \n",
    "    X=X_test_ohe,\n",
    "    y=df_y_test.values,\n",
    "    metric='accuracy',\n",
    "    num_rounds=50,\n",
    "    feature_groups=feature_groups,\n",
    "    seed=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAE6CAYAAAAY+Jn8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAap0lEQVR4nO3dfbAdd33f8fcHGcXGwJiHO8WVBHaIClFTbCuqEbhhzGMlQ1BJobEpceLQqG5sHkozwWQyA+k0MzQEEkyMVQVMcENwgOCOABVTSKGkIJAMjrDsaHojHiwsh2sexINbZJlv/zirnOPLle4e6eru0d73a+aMzv72tzrf85X00d49e3ZTVUiS+ushXRcgSTq5DHpJ6jmDXpJ6zqCXpJ4z6CWp507ruoC5PPaxj61zzjmn6zIk6ZRx66233ltVU3Otm8igP+ecc9i1a1fXZUjSKSPJV4+2zkM3ktRzBr0k9ZxBL0k9Z9BLUs8Z9JLUcwa9JPWcQS9JPWfQS1LPGfSS1HMG/QS7+OKLufjii7suQ9IpzqCXpJ5rFfRJNiTZm2Q6yTVzrE+Sa5v1u5OsHVn375PsSXJ7kvcmOX0h34Ak6djmDfoky4DrgI3AGuCyJGtmTdsIrG4em4Hrm21XAK8E1lXVzwDLgEsXrHpJ0rza7NFfCExX1b6qOgTcBGyaNWcTcGMN7ADOSnJ2s+404IwkpwEPA+5eoNolSS20CfoVwF0jy/ubsXnnVNXXgd8HvgYcAA5W1cfmepEkm5PsSrJrZmambf2SpHm0CfrMMVZt5iR5FIO9/XOBfwicmeRlc71IVW2tqnVVtW5qas5r50uSjkOboN8PrBpZXsmPH3452pznAF+uqpmquh/4IPD04y9XkjSuNkG/E1id5Nwkyxl8mLpt1pxtwOXN2TfrGRyiOcDgkM36JA9LEuDZwJ0LWL8kaR7z3kqwqg4nuRq4hcFZMzdU1Z4kVzbrtwDbgUuAaeA+4Ipm3eeSfAD4AnAY+CKw9WS8EUnS3FrdM7aqtjMI89GxLSPPC7jqKNu+Hnj9CdQoSToBfjNWknrOoJeknjPoJannDHpJ6jmDXpJ6zqCXpJ4z6CWp5wx6Seo5g16Ses6gl6SeM+glqecMeknqOYNeknrOoJeknjPoJannDHpJ6rlWQZ9kQ5K9SaaTXDPH+iS5tlm/O8naZvxJSW4beXw3yasX+D1Iko5h3jtMJVkGXAc8l8FNwHcm2VZVd4xM2wisbh5PBa4HnlpVe4HzR36frwM3L+QbkCQdW5s9+guB6araV1WHgJuATbPmbAJurIEdwFlJzp4159nA31bVV0+4aklL3sUXX8zFF1/cdRmnhDZBvwK4a2R5fzM27pxLgfce7UWSbE6yK8mumZmZFmVJktpoE/SZY6zGmZNkOfBC4P1He5Gq2lpV66pq3dTUVIuyJElttAn6/cCqkeWVwN1jztkIfKGq/u54ipQ04OEKHY82Qb8TWJ3k3GbP/FJg26w524DLm7Nv1gMHq+rAyPrLOMZhG0nSyTPvWTdVdTjJ1cAtwDLghqrak+TKZv0WYDtwCTAN3AdccWT7JA9jcMbOv1348rUUHNmD/eQnP9lpHdKpat6gB6iq7QzCfHRsy8jzAq46yrb3AY85gRolSSfAb8ZKUs8Z9JLUcwa9JPWcQS9JPWfQS1LPGfSS1HMGvST1nEEvST1n0EtSzxn0ktRzBr0k9ZxBL0k9Z9BLUs8Z9JLUcwa9JPWcQS9JPdcq6JNsSLI3yXSSa+ZYnyTXNut3J1k7su6sJB9I8jdJ7kzytIV8A5KkY5s36JMsA65jcIPvNcBlSdbMmrYRWN08NgPXj6x7K/DRqnoycB5w5wLULUlqqc2tBC8EpqtqH0CSm4BNwB0jczYBNza3FNzR7MWfDfwAeAbwKwBVdQg4tHDld+Ocaz6yKK9zz75vLtrrfeWNzz/pryGpG20O3awA7hpZ3t+MtZnzk8AM8K4kX0zyjiRnnkC9kqQxtQn6zDFWLeecBqwFrq+qCxjs4f/YMX6AJJuT7Eqya2ZmpkVZkqQ22gT9fmDVyPJK4O6Wc/YD+6vqc834BxgE/4+pqq1Vta6q1k1NTbWpXZLUQpug3wmsTnJukuXApcC2WXO2AZc3Z9+sBw5W1YGquge4K8mTmnnP5sHH9iVJJ9m8H8ZW1eEkVwO3AMuAG6pqT5Irm/VbgO3AJcA0cB9wxchv8QrgPc1/EvtmrZMknWRtzrqhqrYzCPPRsS0jzwu46ijb3gasO/4SJUknwm/GSlLPGfSS1HMGvST1nEEvST1n0EtSzxn0ktRzBr0k9ZxBL0k9Z9BLUs8Z9JLUcwa9JPWcQS9JPWfQS1LPGfSS1HMGvST1nEEvST1n0EtSz7UK+iQbkuxNMp3kmjnWJ8m1zfrdSdaOrPtKki8luS3JroUsXpI0v3lvJZhkGXAd8FxgP7AzybaqGr3J90ZgdfN4KnB98+sRz6yqexesaklSa2326C8EpqtqX1UdAm4CNs2aswm4sQZ2AGclOXuBa5UkHYc2Qb8CuGtkeX8z1nZOAR9LcmuSzUd7kSSbk+xKsmtmZqZFWZKkNtoEfeYYqzHmXFRVaxkc3rkqyTPmepGq2lpV66pq3dTUVIuyJElttAn6/cCqkeWVwN1t51TVkV+/AdzM4FCQJGmRtAn6ncDqJOcmWQ5cCmybNWcbcHlz9s164GBVHUhyZpJHACQ5E3gecPsC1i9Jmse8Z91U1eEkVwO3AMuAG6pqT5Irm/VbgO3AJcA0cB9wRbP5PwBuTnLktf6sqj664O9CnTnnmo+c9Ne4Z983F+21AL7yxucvyutIi2XeoAeoqu0Mwnx0bMvI8wKummO7fcB5J1ijJOkEtAp6SfPzpxtNKi+BIEk9Z9BLUs8Z9JLUcwa9JPWcH8ZKWnB+MD1Z3KOXpJ4z6CWp5wx6Seo5g16Ses6gl6SeM+glqecMeknqOYNeknrOoJeknjPoJannWgV9kg1J9iaZTnLNHOuT5Npm/e4ka2etX5bki0k+vFCFS5LamTfokywDrgM2AmuAy5KsmTVtI7C6eWwGrp+1/lXAnSdcrSRpbG326C8EpqtqX1UdAm4CNs2aswm4sQZ2AGclORsgyUrg+cA7FrBuSVJLbYJ+BXDXyPL+ZqztnD8EfhP40bFeJMnmJLuS7JqZmWlRliSpjTZBnznGqs2cJC8AvlFVt873IlW1tarWVdW6qampFmVJktpocz36/cCqkeWVwN0t57wYeGGSS4DTgUcm+dOqetnxl7x0PO6lb+y6BEk90GaPfiewOsm5SZYDlwLbZs3ZBlzenH2zHjhYVQeq6nVVtbKqzmm2+0tDXpIW17x79FV1OMnVwC3AMuCGqtqT5Mpm/RZgO3AJMA3cB1xx8kqWJI2j1a0Eq2o7gzAfHdsy8ryAq+b5PT4JfHLsCiVJJ8RvxkpSzxn0ktRzBr0k9ZxBL0k9Z9BLUs8Z9JLUcwa9JPWcQS9JPWfQS1LPGfSS1HMGvST1nEEvST1n0EtSzxn0ktRzBr0k9ZxBL0k91yrok2xIsjfJdJJr5lifJNc263cnWduMn57k80n+OsmeJL+z0G9AknRs8wZ9kmXAdcBGYA1wWZI1s6ZtBFY3j83A9c34D4FnVdV5wPnAhuaespKkRdJmj/5CYLqq9lXVIeAmYNOsOZuAG2tgB3BWkrOb5e83cx7aPGqhipckza9N0K8A7hpZ3t+MtZqTZFmS24BvAP+jqj4314sk2ZxkV5JdMzMzLcuXJM2nTdBnjrHZe+VHnVNVD1TV+cBK4MIkPzPXi1TV1qpaV1XrpqamWpQlSWqjTdDvB1aNLK8E7h53TlV9B/gksGHcIiVJx69N0O8EVic5N8ly4FJg26w524DLm7Nv1gMHq+pAkqkkZwEkOQN4DvA3C1e+JGk+p803oaoOJ7kauAVYBtxQVXuSXNms3wJsBy4BpoH7gCuazc8G3t2cufMQ4H1V9eGFfxuSpKOZN+gBqmo7gzAfHdsy8ryAq+bYbjdwwQnWKEk6Aa2CXurS4176xq5LkE5pXgJBknrOoJeknjPoJannDHpJ6jk/jJVOIX4wrePhHr0k9ZxBL0k9Z9BLUs8Z9JLUcwa9JPWcQS9JPWfQS1LPGfSS1HMGvST1nEEvST3XKuiTbEiyN8l0kmvmWJ8k1zbrdydZ24yvSvI/k9yZZE+SVy30G5AkHdu8Qd/cBvA6YCOwBrgsyZpZ0zYCq5vHZuD6Zvww8B+q6qeB9cBVc2wrSTqJ2lzU7EJguqr2ASS5CdgE3DEyZxNwY3NLwR1JzkpydlUdAA4AVNX3ktwJrJi1rSSNzQu8tdfm0M0K4K6R5f3N2FhzkpzD4P6xnxu7SknScWsT9JljrMaZk+ThwF8Ar66q7875IsnmJLuS7JqZmWlRliSpjTZBvx9YNbK8Eri77ZwkD2UQ8u+pqg8e7UWqamtVrauqdVNTU21qlyS10CbodwKrk5ybZDlwKbBt1pxtwOXN2TfrgYNVdSBJgHcCd1bVWxa0cklSK/N+GFtVh5NcDdwCLANuqKo9Sa5s1m8BtgOXANPAfcAVzeYXAb8EfCnJbc3Yb1XV9gV9F5Kko2p1K8EmmLfPGtsy8ryAq+bY7q+Y+/i9JGmR+M1YSeo5g16Ses6gl6SeM+glqecMeknqOYNeknrOoJeknjPoJannDHpJ6jmDXpJ6zqCXpJ4z6CWp5wx6Seo5g16Ses6gl6SeM+glqedaBX2SDUn2JplOcs0c65Pk2mb97iRrR9bdkOQbSW5fyMIlSe3MG/RJlgHXARuBNcBlSdbMmrYRWN08NgPXj6z7E2DDQhQrSRpfmz36C4HpqtpXVYeAm4BNs+ZsAm6sgR3AWUnOBqiq/wV8ayGLliS11yboVwB3jSzvb8bGnSNJ6kCboJ/r5t51HHOO/SLJ5iS7kuyamZkZZ1NJ0jG0Cfr9wKqR5ZXA3ccx55iqamtVrauqdVNTU+NsKkk6hjZBvxNYneTcJMuBS4Fts+ZsAy5vzr5ZDxysqgMLXKsk6TjMG/RVdRi4GrgFuBN4X1XtSXJlkiubaduBfcA08MfArx/ZPsl7gc8CT0qyP8nLF/g9SJKO4bQ2k6pqO4MwHx3bMvK8gKuOsu1lJ1KgJOnE+M1YSeo5g16Ses6gl6SeM+glqecMeknqOYNeknrOoJeknjPoJannDHpJ6jmDXpJ6zqCXpJ4z6CWp5wx6Seo5g16Ses6gl6SeM+glqecMeknquVZBn2RDkr1JppNcM8f6JLm2Wb87ydq220qSTq55gz7JMuA6YCOwBrgsyZpZ0zYCq5vHZuD6MbaVJJ1EbfboLwSmq2pfVR0CbgI2zZqzCbixBnYAZyU5u+W2kqSTqM3NwVcAd40s7wee2mLOipbbApBkM4OfBgC+n2Rvi9qWgscC957sF8l/PtmvcMIWpQ9gL0bZi6FToBdPONqKNkGfOcaq5Zw22w4Gq7YCW1vUs6Qk2VVV67quo2v2YcheDNmLdtoE/X5g1cjySuDulnOWt9hWknQStTlGvxNYneTcJMuBS4Fts+ZsAy5vzr5ZDxysqgMtt5UknUTz7tFX1eEkVwO3AMuAG6pqT5Irm/VbgO3AJcA0cB9wxbG2PSnvpL88nDVgH4bsxZC9aCFVcx4ylyT1hN+MlaSeM+glqecMeknqOYNeknquzXn0WgRJ3sZRvkwGUFWvXMRyJPWIe/STYxdw6zEeApJ8qesaFlOSVUluSvLpJL+V5KEj6/5bh6UtuiRPTvLfk3wkyROT/EmS7yT5fJKf7rq+SeYe/YSoqnd3XcOkSPILR1sFPG4xa5kANwB/AewAXg58KsnPV9U3Oca1TXpqK/Am4OHAXwKvZfCdnRcAfwQ8u7vSJpvn0U+YJFMM/gKvAU4/Ml5Vz+qsqEWW5H7gPcx9KOvFVfWIRS6pM0luq6rzR5ZfBrwOeCHw/qpae7Rt+ybJF6vqgub5dFX91Mi6LyylXozLPfrJ8x7gz4HnA1cCvwzMdFrR4tsN/H5V3T57RZLndFBPlx6a5PSq+n8AVfWnSe5h8G3zM7stbdEtG3n+llnrli9mIacaj9FPnsdU1TuB+6vqU1X1q8D6rotaZK8GvnuUdS9axDomwTuYdWnvqvo48BLgx/4j7LnrkjwcoKrefmQwyU8BH++sqlOAQT957m9+PZDk+UkuYHDVzyWjqj5dVV9LctEcq39i0QvqUFX9QVV9anYvquqLwBu6qaobVfVfqur7c/RiGnh/R2WdEjxGP2GSvAD4NIPLO78NeCTwO1W15K76Oddx16V6LNZeDNmL8XmMfsJU1YebpweBZ3ZZS1eSPA14OjCV5DUjqx7Jg4/T9p69GLIXx89DNxMmybuTnDWy/KgkN3RYUheWMziF7jTgESOP7wIv7rCuLtiLIXtxnDx0M2FGTyE71thSkOQJVfXVruuYBPZiyF6Mz0M3k+chSR5VVd8GSPJolu6f008k2Qqcw0gPltJ3CkbYiyF7MaalGiCT7M3AZ5J8oFl+CfC7HdbTpfcDWxicYvhAx7V0zV4M2YsxeehmAiVZAzyLwVf+P1FVd3RcUieS3FpVP9t1HZPAXgzZi/EZ9BMiySOr6rvNoZofU1XfWuyaupbkDcA3gJuBHx4Ztxf2AnsxFoN+QiT5cFW9IMmXefA1XgJUVf1kR6V1punFbPZiyF4MLcletGXQT5AkAVZV1de6rkVSf3ge/QSpwf+6N3ddx6RI8rAkv92cYUGS1c03h5ccezFkL8Zn0E+eHUn+addFTIh3AYcYfBsSYD/wn7orp1P2YshejMmgnzzPBD6b5G+T7E7ypSS7uy6qI0+sqt+judBbVf1fBp9ZLEX2YshejMnz6CfPxq4LmCCHkpxB8+F0kicycpbFEmMvhuzFmAz6CVNVX01yHvBzzdCnq+qvu6ypQ68HPgqsSvIe4CLgVzqtqDv2YshejMmzbiZMklcBvwZ8sBl6EbC1qt7WXVXdSfIYBjdeCbCjqu7tuKTO2IshezEeg37CNMfjn1ZVP2iWzwQ+W1VP6baybiR5Cj9+TZMPHnWDHrMXQ/ZiPB66mTzhwdfveIAl+kFTc3nmpwB7gB81w8Xwp50lw14M2YvxGfST513A55IcOZ/+XwDv7K6cTq2vqjVdFzEh7MWQvRiTp1dOmKp6C3AF8C3g28AVVfWHnRbVnc82F3iTvRhlL8bkMfoJc5SLmn2vqu6fY7zXkjwD+BBwD4PT545c92fJfV5hL4bsxfgM+gmT5CsMbgz+bQZ/gc8CDjC4Wt+vVdWtnRW3yJJMA68BvsTwWCxL8e5C9mLIXozPY/ST56PAzVV1C0CS5wEbgPcBbwee2mFti+1rVbWt6yImhL0Yshdjco9+wiTZVVXr5hpLcltVnd9RaYsuydsZ/ETzIR583fEld3aFvRiyF+Nzj37yfCvJa4GbmuVfBL6dZBkjP6YuEWcw+If8vJGxpXoanb0Yshdjco9+wiR5LIOveP+zZuivgP8IHAQeX1XTXdUm6dTk6ZUTpqrurapXAD9XVRdU1SuqaqaqDi21kE/yj5J8IsntzfJTkvx213V1wV4M2YvxGfQTJsnTk9wB3NEsn9cck1yK/hh4HcPL0e4GLu20ou7YiyF7MSaDfvL8AfDPgW8CNFeufEanFXXnYVX1+VljhzuppHv2YshejMmgn0BVddesoQfmnNh/9zbXGj9y3fEXM/hOwVJkL4bsxZg862by3JXk6UAlWQ68Eriz45q6chWwFXhykq8DXwb+dbcldcZeDNmLMRn0k+dK4K3ACgb3wvwY8OudVtSB5nTSf1dVz2ku1fyQqvpe13V1wV4M2YvjY9BPnidV1YP2TpJcBPzvjurpRFU9kORnm+c/6LqeLtmLIXtxfDyPfsIk+UJVrZ1vbClI8mZgNfB+4O//US/Fb0DaiyF7MT736CdEkqcBTwemkrxmZNUjgWXdVNW5RzM4++hZI2NL9RuQ9mLIXozJoJ8cy4GHM/gzecTI+HeBF3dSUceq6oqua5gU9mLIXozPQzcTJskTvNzqQJJ30ZxCN6qqfrWDcjplL4bsxfjco5889yV5E/CPgdOPDFbVs46+SW99eOT56cCLgLs7qqVr9mLIXozJPfoJk+RjwJ8Dv8HgVMtfBmaq6rWdFjYBkjwE+PgS/U/vQezFkL2Yn9+MnTyPqap3AvdX1aeaH0fXd13UhFgNPL7rIiaEvRiyF/Pw0M3kOXJv2ANJns/gR9KVHdbTmSTf48HHYu8BluRPNvZiyF6Mz0M3EybJC4BPM7hv7NsYnF75hqr6UKeFSTpleehm8ryEwX/At1fVM4HnMviwaclJclHzNXeSvCzJW5I8oeu6umAvhuzF+Az6yfOUqvrOkYWq+hZwQXfldOp6BmchnQf8JvBV4MZuS+qMvRiyF2My6CfPQ5I86shCkkezdD9LOVyDY4ubgLdW1Vt58JfJlhJ7MWQvxrRUA2SSvRn4TJIPMPjA6V8Bv9ttSZ35XpLXAS8DntFcufChHdfUFXsxZC/G5B79hKmqG4F/CfwdMAP8QlX9126r6swvAj8EXl5V9zC4dPObui2pM/ZiyF6MybNuJKnn3KPXxEqyPsnOJN9PcijJA0kOdl1XF+zFkL0Yn0GvSfZHwGXA/wHOAP4NcF2nFXXHXgzZizH5YawmWlVNJ1lWVQ8A70ryma5r6oq9GLIX4zHoNcnua26QfluS3wMOAGd2XFNX7MWQvRiTh240yX6Jwd/RqxncMm4VgzOSliJ7MWQvxuRZN5poSc4AHl9Ve7uupWv2YshejMc9ek2sJD8P3AZ8tFk+P8m2TovqiL0YshfjM+g1yd4AXAh8B6CqbgPO6ayabr0Be3HEG7AXYzHoNckOV5XnRw/YiyF7MSbPutEkuz3JS4FlSVYDrwSW6ml09mLIXozJPXpNslcwuEn6D4E/Aw4Cr+q0ou7YiyF7MSaDXpNsTfM4DTidwWVpd3ZaUXfsxZC9GJOnV2piJdkL/AZwO/CjI+NV9dXOiuqIvRiyF+PzGL0m2Yz3yv179mLIXozJPXpNrCTPZnDxqk8wOB4LQFV9sLOiOmIvhuzF+Nyj1yS7Angyg7sHHfkRvYCl+A/aXgzZizEZ9Jpk51XVP+m6iAlhL4bsxZg860aTbEeSNV0XMSHsxZC9GJPH6DWxktwJPBH4MoNjsQGqqp7SaWEdsBdD9mJ8Br0mVpInzDW+FE+jsxdD9mJ8Br0k9ZzH6CWp5wx6Seo5g16Ses6gl6Se+/89RLWid2ro9QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "feature_names = ['measurement1', 'measurement2', 'measurement3', 'categorical']\n",
    "\n",
    "std = np.std(imp_all, axis=1)\n",
    "indices = np.argsort(imp_vals)[::-1]\n",
    "\n",
    "plt.figure()\n",
    "plt.bar(range(len(feature_names)), imp_vals[indices],\n",
    "        yerr=std[indices])\n",
    "plt.xticks(range(len(feature_names)),\n",
    "           np.array(feature_names)[indices], rotation=90)\n",
    "plt.xlim([-1, len(feature_names)])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "## feature_importance_permutation\n",
      "\n",
      "*feature_importance_permutation(X, y, predict_method, metric, num_rounds=1, feature_groups=None, seed=None)*\n",
      "\n",
      "Feature importance imputation via permutation importance\n",
      "\n",
      "**Parameters**\n",
      "\n",
      "\n",
      "- `X` : NumPy array, shape = [n_samples, n_features]\n",
      "\n",
      "    Dataset, where n_samples is the number of samples and\n",
      "    n_features is the number of features.\n",
      "\n",
      "\n",
      "- `y` : NumPy array, shape = [n_samples]\n",
      "\n",
      "    Target values.\n",
      "\n",
      "\n",
      "- `predict_method` : prediction function\n",
      "\n",
      "    A callable function that predicts the target values\n",
      "    from X.\n",
      "\n",
      "\n",
      "- `metric` : str, callable\n",
      "\n",
      "    The metric for evaluating the feature importance through\n",
      "    permutation. By default, the strings 'accuracy' is\n",
      "    recommended for classifiers and the string 'r2' is\n",
      "    recommended for regressors. Optionally, a custom\n",
      "    scoring function (e.g., `metric=scoring_func`) that\n",
      "    accepts two arguments, y_true and y_pred, which have\n",
      "    similar shape to the `y` array.\n",
      "\n",
      "\n",
      "- `num_rounds` : int (default=1)\n",
      "\n",
      "    Number of rounds the feature columns are permuted to\n",
      "    compute the permutation importance.\n",
      "\n",
      "\n",
      "- `feature_groups` : list or None (default=None)\n",
      "\n",
      "    Optional argument for treating certain features as a group.\n",
      "    For example `[1, 2, [3, 4, 5]]`, which can be useful for\n",
      "    interpretability, for example, if features 3, 4, 5 are one-hot\n",
      "    encoded features.\n",
      "\n",
      "\n",
      "- `seed` : int or None (default=None)\n",
      "\n",
      "    Random seed for permuting the feature columns.\n",
      "\n",
      "**Returns**\n",
      "\n",
      "\n",
      "- `mean_importance_vals, all_importance_vals` : NumPy arrays.\n",
      "\n",
      "    The first array, mean_importance_vals has shape [n_features, ] and\n",
      "    contains the importance values for all features.\n",
      "    The shape of the second array is [n_features, num_rounds] and contains\n",
      "    the feature importance for each repetition. If num_rounds=1,\n",
      "    it contains the same values as the first array, mean_importance_vals.\n",
      "\n",
      "**Examples**\n",
      "\n",
      "For usage examples, please see\n",
      "    https://rasbt.github.io/mlxtend/user_guide/evaluate/feature_importance_permutation/\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open('../../api_modules/mlxtend.evaluate/feature_importance_permutation.md', 'r') as f:\n",
    "    s = f.read() \n",
    "print(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
